[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:18:50 dist-test-slave-d3n8 krb5kdc[4709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:18:50 dist-test-slave-d3n8 krb5kdc[4709](info): set up 2 sockets
Jun 19 08:18:50 dist-test-slave-d3n8 krb5kdc[4709](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 08:18:51.670737 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:18:51 dist-test-slave-d3n8 krb5kdc[4709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:18:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:18:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:18:51.804960 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:42267
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:42267
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:18:51.888356 4722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:18:51.888625 4722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:18:51.888664 4722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:18:51.889964 4722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:18:51.890048 4722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:18:51.890079 4722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:18:51.890108 4722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:18:51.890163 4722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:18:51.891805 4722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:42267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:42267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:18:51.892110 4722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:18:51.892365 4722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:18:51.898444 4730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:51.898416 4727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:51.898375 4728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:18:51.898466 4722 server_base.cc:1034] running on GCE node
I20260619 08:18:51.898844 4722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:18:51.899178 4722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:18:51.900363 4722 hybrid_clock.cc:648] HybridClock initialized: now 1781857131900348 us; error 56 us; skew 500 ppm
Jun 19 08:18:51 dist-test-slave-d3n8 krb5kdc[4709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857131, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:18:51.903119 4722 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:18:51.903541 4722 webserver.cc:460] Webserver started at http://127.4.151.190:36009/ using document root <none> and password file <none>
I20260619 08:18:51.903733 4722 fs_manager.cc:359] Metadata directory not provided
I20260619 08:18:51.903774 4722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:18:51.903868 4722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:18:51.904946 4722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "1cdc74e3cc6e4fca978ee852a5d4d677"
format_stamp: "Formatted at 2026-06-19 08:18:51 on dist-test-slave-d3n8"
server_key: "c9c61b074d9da67f2fef7d2183743744"
server_key_iv: "e7fbf800437f7107a5e0cd9393fc0666"
server_key_version: "encryptionkey@0"
I20260619 08:18:51.905287 4722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "1cdc74e3cc6e4fca978ee852a5d4d677"
format_stamp: "Formatted at 2026-06-19 08:18:51 on dist-test-slave-d3n8"
server_key: "c9c61b074d9da67f2fef7d2183743744"
server_key_iv: "e7fbf800437f7107a5e0cd9393fc0666"
server_key_version: "encryptionkey@0"
I20260619 08:18:51.906781 4722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:18:51.907570 4737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:51.907797 4722 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:18:51.907894 4722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "1cdc74e3cc6e4fca978ee852a5d4d677"
format_stamp: "Formatted at 2026-06-19 08:18:51 on dist-test-slave-d3n8"
server_key: "c9c61b074d9da67f2fef7d2183743744"
server_key_iv: "e7fbf800437f7107a5e0cd9393fc0666"
server_key_version: "encryptionkey@0"
I20260619 08:18:51.907982 4722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:18:51.930673 4722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:18:51.933007 4722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:18:51.933133 4722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:18:51.937223 4722 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:42267
I20260619 08:18:51.937316 4789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:42267 every 8 connection(s)
I20260619 08:18:51.937719 4722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:18:51.938683 4790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:18:51.940688 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 4722
I20260619 08:18:51.940804 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:18:51.941493 4790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cdc74e3cc6e4fca978ee852a5d4d677: Bootstrap starting.
I20260619 08:18:51.941612 4702 external_mini_cluster.cc:1428] Setting key e3ec312d67b78c5505c5570ba95e1d6e
I20260619 08:18:51.942232 4790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cdc74e3cc6e4fca978ee852a5d4d677: Neither blocks nor log segments found. Creating new log.
I20260619 08:18:51.942605 4790 log.cc:824] T 00000000000000000000000000000000 P 1cdc74e3cc6e4fca978ee852a5d4d677: Log is configured to *not* fsync() on all Append() calls
I20260619 08:18:51.943409 4790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cdc74e3cc6e4fca978ee852a5d4d677: No bootstrap required, opened a new log
I20260619 08:18:51.945856 4790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cdc74e3cc6e4fca978ee852a5d4d677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cdc74e3cc6e4fca978ee852a5d4d677" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42267 } }
I20260619 08:18:51.946043 4790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cdc74e3cc6e4fca978ee852a5d4d677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:18:51.946090 4790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cdc74e3cc6e4fca978ee852a5d4d677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cdc74e3cc6e4fca978ee852a5d4d677, State: Initialized, Role: FOLLOWER
I20260619 08:18:51.946233 4790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cdc74e3cc6e4fca978ee852a5d4d677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cdc74e3cc6e4fca978ee852a5d4d677" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42267 } }
I20260619 08:18:51.946334 4790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cdc74e3cc6e4fca978ee852a5d4d677 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:18:51.946381 4790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cdc74e3cc6e4fca978ee852a5d4d677 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:18:51.946432 4790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cdc74e3cc6e4fca978ee852a5d4d677 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:18:51.947019 4790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cdc74e3cc6e4fca978ee852a5d4d677 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cdc74e3cc6e4fca978ee852a5d4d677" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42267 } }
I20260619 08:18:51.947151 4790 leader_election.cc:302] T 00000000000000000000000000000000 P 1cdc74e3cc6e4fca978ee852a5d4d677 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cdc74e3cc6e4fca978ee852a5d4d677; no voters:
I20260619 08:18:51.947361 4790 leader_election.cc:288] T 00000000000000000000000000000000 P 1cdc74e3cc6e4fca978ee852a5d4d677 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:18:51.947427 4795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cdc74e3cc6e4fca978ee852a5d4d677 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:18:51.947579 4795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cdc74e3cc6e4fca978ee852a5d4d677 [term 1 LEADER]: Becoming Leader. State: Replica: 1cdc74e3cc6e4fca978ee852a5d4d677, State: Running, Role: LEADER
I20260619 08:18:51.947638 4790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cdc74e3cc6e4fca978ee852a5d4d677 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:18:51.947681 4795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cdc74e3cc6e4fca978ee852a5d4d677 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cdc74e3cc6e4fca978ee852a5d4d677" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42267 } }
I20260619 08:18:51.948025 4795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cdc74e3cc6e4fca978ee852a5d4d677 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cdc74e3cc6e4fca978ee852a5d4d677. Latest consensus state: current_term: 1 leader_uuid: "1cdc74e3cc6e4fca978ee852a5d4d677" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cdc74e3cc6e4fca978ee852a5d4d677" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42267 } } }
I20260619 08:18:51.948112 4795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cdc74e3cc6e4fca978ee852a5d4d677 [sys.catalog]: This master's current role is: LEADER
I20260619 08:18:51.948791 4802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:18:51.949342 4802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:18:51.948038 4796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cdc74e3cc6e4fca978ee852a5d4d677 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cdc74e3cc6e4fca978ee852a5d4d677" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cdc74e3cc6e4fca978ee852a5d4d677" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42267 } } }
I20260619 08:18:51.949528 4796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cdc74e3cc6e4fca978ee852a5d4d677 [sys.catalog]: This master's current role is: LEADER
I20260619 08:18:51.950919 4802 catalog_manager.cc:1347] Generated new cluster ID: ec005a05d2f54ac6b63179b2fdae6d08
I20260619 08:18:51.950976 4802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:18:51 dist-test-slave-d3n8 krb5kdc[4709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:18:51.959501 4802 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:18:51.959641 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:51.942663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43098 (local address 127.4.151.190:42267)
0619 08:18:51.942873 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:51.942883 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:51.943166 (+ 283us) server_negotiation.cc:407] Connection header received
0619 08:18:51.945802 (+ 2636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:51.945810 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:51.945872 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:51.946026 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:51.948153 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:51.949616 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:51.950944 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:51.951393 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:51.951474 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:51.955486 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:51.955491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:51.955491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:51.955503 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:51.957500 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:51.958038 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:51.958041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:51.958042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:51.958147 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:51.958488 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:51.958489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:51.958489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:51.958728 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:51.958907 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:51.959215 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 08:18:51.959352 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":97,"threads_started":1}
I20260619 08:18:51.960068 4802 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:18:51.965982 4802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cdc74e3cc6e4fca978ee852a5d4d677: Generated new TSK 0
I20260619 08:18:51.966162 4802 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.151.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
I20260619 08:18:52.001899 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:42267
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:18:52.086236 4816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:18:52.086454 4816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:18:52.086479 4816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:18:52.087654 4816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:18:52.087702 4816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:18:52.087730 4816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:18:52.089233 4816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:42267
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:18:52.089515 4816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:18:52.089764 4816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:18:52.094851 4821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:52.094821 4822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:52.094858 4824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:18:52.095191 4816 server_base.cc:1034] running on GCE node
I20260619 08:18:52.095357 4816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:18:52.095557 4816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:18:52.096709 4816 hybrid_clock.cc:648] HybridClock initialized: now 1781857132096695 us; error 28 us; skew 500 ppm
Jun 19 08:18:52 dist-test-slave-d3n8 krb5kdc[4709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857132, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:18:52.099475 4816 init.cc:377] Logged in from keytab as kudu/127.4.151.129@KRBTEST.COM (short username kudu)
I20260619 08:18:52.099961 4816 webserver.cc:460] Webserver started at http://127.4.151.129:44789/ using document root <none> and password file <none>
I20260619 08:18:52.100210 4816 fs_manager.cc:359] Metadata directory not provided
I20260619 08:18:52.100255 4816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:18:52.100383 4816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:18:52.101513 4816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "e6231428f6064ea7a7ccc19bae2186b4"
format_stamp: "Formatted at 2026-06-19 08:18:52 on dist-test-slave-d3n8"
server_key: "81107c2ccee66a05f4a46fc63768b7b5"
server_key_iv: "ffe36f3865e1447d1ce5780b94a2e93d"
server_key_version: "encryptionkey@0"
I20260619 08:18:52.101878 4816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "e6231428f6064ea7a7ccc19bae2186b4"
format_stamp: "Formatted at 2026-06-19 08:18:52 on dist-test-slave-d3n8"
server_key: "81107c2ccee66a05f4a46fc63768b7b5"
server_key_iv: "ffe36f3865e1447d1ce5780b94a2e93d"
server_key_version: "encryptionkey@0"
I20260619 08:18:52.103251 4816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:18:52.104085 4831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:52.104281 4816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:18:52.104389 4816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "e6231428f6064ea7a7ccc19bae2186b4"
format_stamp: "Formatted at 2026-06-19 08:18:52 on dist-test-slave-d3n8"
server_key: "81107c2ccee66a05f4a46fc63768b7b5"
server_key_iv: "ffe36f3865e1447d1ce5780b94a2e93d"
server_key_version: "encryptionkey@0"
I20260619 08:18:52.104471 4816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:18:52.126964 4816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:18:52.129294 4816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:18:52.129557 4816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:18:52.129812 4816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:18:52.130147 4816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:18:52.130203 4816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:52.130249 4816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:18:52.130272 4816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:52.136391 4816 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:34719
I20260619 08:18:52.136436 4944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:34719 every 8 connection(s)
I20260619 08:18:52.136754 4816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
I20260619 08:18:52.137061 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 4816
I20260619 08:18:52.137154 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:18:52.137292 4702 external_mini_cluster.cc:1428] Setting key ab3a5606e4cc402fde8e45ec1d429d9f
Jun 19 08:18:52 dist-test-slave-d3n8 krb5kdc[4709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857132, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.151.130@KRBTEST.COM; defaulting to no policy
I20260619 08:18:52.147850 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:52.137791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:42009 (local address 127.4.151.190:42267)
0619 08:18:52.137868 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:52.137871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:52.138159 (+ 288us) server_negotiation.cc:407] Connection header received
0619 08:18:52.138759 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:52.138761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:52.138773 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:52.138820 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:52.139817 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:52.140331 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:52.141904 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:52.142175 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:52.142247 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:52.144744 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:52.144745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:52.144746 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:52.144754 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:52.146283 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:52.146755 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:52.146756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:52.146756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:52.146810 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:52.147198 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:52.147198 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:52.147199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:52.147310 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:52.147367 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:52.147693 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 08:18:52.147778 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 08:18:52.148147 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:52.137799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42267 (local address 127.4.151.129:42009)
0619 08:18:52.137969 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:52.138036 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:52.138640 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:52.138896 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:52.138899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:52.139506 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:18:52.139710 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:52.139714 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:52.140437 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:52.140438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:52.141789 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:52.141792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:52.142278 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:52.142279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:52.142440 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:52.142994 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:18:52.143004 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:18:52.144640 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:18:52.146395 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:52.146403 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:52.146405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:52.146684 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:52.146956 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:52.146957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:52.146957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:52.147128 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:52.147374 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:18:52.147379 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:18:52.147574 (+ 195us) client_negotiation.cc:769] Sending connection context
0619 08:18:52.147659 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 08:18:52.147824 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":118,"threads_started":1}
I20260619 08:18:52.148837 4945 heartbeater.cc:347] Connected to a master server at 127.4.151.190:42267
I20260619 08:18:52.148941 4945 heartbeater.cc:464] Registering TS with master...
I20260619 08:18:52.149149 4945 heartbeater.cc:511] Master 127.4.151.190:42267 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.130@KRBTEST.COM" created.
I20260619 08:18:52.149758 4754 ts_manager.cc:194] Registered new tserver with Master: e6231428f6064ea7a7ccc19bae2186b4 (127.4.151.129:34719)
I20260619 08:18:52.150377 4754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.129@KRBTEST.COM'} at 127.4.151.129:42009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
I20260619 08:18:52.164870 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:42267
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:18:52.247978 4950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:18:52.248188 4950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:18:52.248212 4950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:18:52.249521 4950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:18:52.249563 4950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:18:52.249590 4950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:18:52.250988 4950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:42267
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:18:52.251179 4950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:18:52.251349 4950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:18:52.256657 4958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:52.256675 4955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:52.256675 4956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:18:52.256693 4950 server_base.cc:1034] running on GCE node
I20260619 08:18:52.257186 4950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:18:52.257516 4950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:18:52.258720 4950 hybrid_clock.cc:648] HybridClock initialized: now 1781857132258706 us; error 30 us; skew 500 ppm
Jun 19 08:18:52 dist-test-slave-d3n8 krb5kdc[4709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857132, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:18:52.261511 4950 init.cc:377] Logged in from keytab as kudu/127.4.151.130@KRBTEST.COM (short username kudu)
I20260619 08:18:52.261996 4950 webserver.cc:460] Webserver started at http://127.4.151.130:40687/ using document root <none> and password file <none>
I20260619 08:18:52.262238 4950 fs_manager.cc:359] Metadata directory not provided
I20260619 08:18:52.262308 4950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:18:52.262508 4950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:18:52.263599 4950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "0be033246be2416287907db5c73032de"
format_stamp: "Formatted at 2026-06-19 08:18:52 on dist-test-slave-d3n8"
server_key: "cd5092e2c49aa3794e4bbe5634d4adeb"
server_key_iv: "5f0fd92e0b37e1b77e0a7236b243b3a0"
server_key_version: "encryptionkey@0"
I20260619 08:18:52.263972 4950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "0be033246be2416287907db5c73032de"
format_stamp: "Formatted at 2026-06-19 08:18:52 on dist-test-slave-d3n8"
server_key: "cd5092e2c49aa3794e4bbe5634d4adeb"
server_key_iv: "5f0fd92e0b37e1b77e0a7236b243b3a0"
server_key_version: "encryptionkey@0"
I20260619 08:18:52.265354 4950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:18:52.266413 4965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:52.266647 4950 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:18:52.266752 4950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "0be033246be2416287907db5c73032de"
format_stamp: "Formatted at 2026-06-19 08:18:52 on dist-test-slave-d3n8"
server_key: "cd5092e2c49aa3794e4bbe5634d4adeb"
server_key_iv: "5f0fd92e0b37e1b77e0a7236b243b3a0"
server_key_version: "encryptionkey@0"
I20260619 08:18:52.266844 4950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:18:52.292695 4950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:18:52.295490 4950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:18:52.295660 4950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:18:52.295912 4950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:18:52.296237 4950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:18:52.296299 4950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:52.296335 4950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:18:52.296368 4950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:52.302628 4950 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:37899
I20260619 08:18:52.302675 5078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:37899 every 8 connection(s)
I20260619 08:18:52.303001 4950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
Jun 19 08:18:52 dist-test-slave-d3n8 krb5kdc[4709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857132, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:18:52.309873 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 4950
I20260619 08:18:52.310027 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:18:52.310235 4702 external_mini_cluster.cc:1428] Setting key e77ab8c8eeb089536461947c1efe87c1
I20260619 08:18:52.313577 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:52.303761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:33043 (local address 127.4.151.190:42267)
0619 08:18:52.303935 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:52.303938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:52.304058 (+ 120us) server_negotiation.cc:407] Connection header received
0619 08:18:52.304595 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:52.304596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:52.304608 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:52.304655 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:52.305686 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:52.306196 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:52.307202 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:52.307485 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:52.307569 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:52.310103 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:52.310105 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:52.310105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:52.310114 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:52.311869 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:52.312430 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:52.312432 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:52.312432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:52.312502 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:52.312741 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:52.312742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:52.312742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:52.312894 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:52.312966 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:52.313296 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 08:18:52.313437 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 08:18:52.313683 5081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:52.303736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42267 (local address 127.4.151.130:33043)
0619 08:18:52.303911 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:52.303961 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:52.304481 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:52.304722 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:52.304726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:52.305307 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:18:52.305580 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:52.305585 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:52.306305 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:52.306306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:52.307099 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:52.307101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:52.307579 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:52.307580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:52.307704 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:52.308331 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:18:52.308339 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:18:52.309970 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:18:52.311954 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:52.311961 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:52.311964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:52.312297 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:52.312572 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:52.312572 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:52.312573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:52.312677 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:52.313011 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:18:52.313017 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:18:52.313139 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 08:18:52.313242 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 08:18:52.313462 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":79,"threads_started":1}
I20260619 08:18:52.314170 5079 heartbeater.cc:347] Connected to a master server at 127.4.151.190:42267
I20260619 08:18:52.314281 5079 heartbeater.cc:464] Registering TS with master...
I20260619 08:18:52.314469 5079 heartbeater.cc:511] Master 127.4.151.190:42267 requested a full tablet report, sending...
I20260619 08:18:52.314853 4754 ts_manager.cc:194] Registered new tserver with Master: 0be033246be2416287907db5c73032de (127.4.151.130:37899)
I20260619 08:18:52.315254 4754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.130@KRBTEST.COM'} at 127.4.151.130:33043
WARNING: no policy specified for kudu/127.4.151.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
I20260619 08:18:52.337280 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:42267
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:18:52.419620 5084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:18:52.419816 5084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:18:52.419840 5084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:18:52.420913 5084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:18:52.420950 5084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:18:52.420976 5084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:18:52.422489 5084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:42267
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:18:52.422676 5084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:18:52.422847 5084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:18:52.428035 5092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:52.428061 5090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:52.428076 5089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:18:52.428169 5084 server_base.cc:1034] running on GCE node
I20260619 08:18:52.428524 5084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:18:52.428742 5084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:18:52.429886 5084 hybrid_clock.cc:648] HybridClock initialized: now 1781857132429834 us; error 66 us; skew 500 ppm
Jun 19 08:18:52 dist-test-slave-d3n8 krb5kdc[4709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857132, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:18:52.432471 5084 init.cc:377] Logged in from keytab as kudu/127.4.151.131@KRBTEST.COM (short username kudu)
I20260619 08:18:52.433059 5084 webserver.cc:460] Webserver started at http://127.4.151.131:37671/ using document root <none> and password file <none>
I20260619 08:18:52.433357 5084 fs_manager.cc:359] Metadata directory not provided
I20260619 08:18:52.433487 5084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:18:52.433655 5084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:18:52.434748 5084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "45d963b39d354e8a946778c8b8e88099"
format_stamp: "Formatted at 2026-06-19 08:18:52 on dist-test-slave-d3n8"
server_key: "399cb01ba372d3ad5b79468fa7906f97"
server_key_iv: "760403994bf244306959c0e9263e83b7"
server_key_version: "encryptionkey@0"
I20260619 08:18:52.435135 5084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "45d963b39d354e8a946778c8b8e88099"
format_stamp: "Formatted at 2026-06-19 08:18:52 on dist-test-slave-d3n8"
server_key: "399cb01ba372d3ad5b79468fa7906f97"
server_key_iv: "760403994bf244306959c0e9263e83b7"
server_key_version: "encryptionkey@0"
I20260619 08:18:52.436478 5084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:18:52.437589 5099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:52.437793 5084 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:18:52.437915 5084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "45d963b39d354e8a946778c8b8e88099"
format_stamp: "Formatted at 2026-06-19 08:18:52 on dist-test-slave-d3n8"
server_key: "399cb01ba372d3ad5b79468fa7906f97"
server_key_iv: "760403994bf244306959c0e9263e83b7"
server_key_version: "encryptionkey@0"
I20260619 08:18:52.437995 5084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:18:52.465602 5084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:18:52.468109 5084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:18:52.468277 5084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:18:52.468531 5084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:18:52.468853 5084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:18:52.468909 5084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:52.468984 5084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:18:52.469014 5084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:52.475144 5084 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:36797
I20260619 08:18:52.475226 5212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:36797 every 8 connection(s)
I20260619 08:18:52.475569 5084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
Jun 19 08:18:52 dist-test-slave-d3n8 krb5kdc[4709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857132, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:18:52.482693 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 5084
I20260619 08:18:52.482762 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:18:52.482950 4702 external_mini_cluster.cc:1428] Setting key 13b69a318958f98771536ca58dba45bd
I20260619 08:18:52.485689 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:52.476322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:41975 (local address 127.4.151.190:42267)
0619 08:18:52.476396 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:52.476398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:52.476609 (+ 211us) server_negotiation.cc:407] Connection header received
0619 08:18:52.477151 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:52.477152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:52.477163 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:52.477224 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:52.478306 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:52.478822 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:52.479983 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:52.480201 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:52.480294 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:52.482697 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:52.482699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:52.482699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:52.482709 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:52.484451 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:52.484837 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:52.484838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:52.484838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:52.484899 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:52.485098 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:52.485099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:52.485099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:52.485211 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:52.485263 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:52.485556 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 08:18:52.485624 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 08:18:52.485901 5215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:52.476299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42267 (local address 127.4.151.131:41975)
0619 08:18:52.476489 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:52.476525 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:52.477043 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:52.477297 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:52.477300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:52.477990 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:18:52.478186 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:52.478191 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:52.478939 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:52.478940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:52.479855 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:52.479858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:52.480305 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:52.480306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:52.480440 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:52.480966 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:18:52.480983 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:18:52.482587 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:18:52.484540 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:52.484545 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:52.484547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:52.484763 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:52.484962 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:52.484963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:52.484963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:52.485046 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:52.485258 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:18:52.485261 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:18:52.485370 (+ 109us) client_negotiation.cc:769] Sending connection context
0619 08:18:52.485526 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 08:18:52.485650 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":79,"threads_started":1}
I20260619 08:18:52.486687 5213 heartbeater.cc:347] Connected to a master server at 127.4.151.190:42267
I20260619 08:18:52.486820 5213 heartbeater.cc:464] Registering TS with master...
I20260619 08:18:52.487026 5213 heartbeater.cc:511] Master 127.4.151.190:42267 requested a full tablet report, sending...
I20260619 08:18:52.487479 4754 ts_manager.cc:194] Registered new tserver with Master: 45d963b39d354e8a946778c8b8e88099 (127.4.151.131:36797)
I20260619 08:18:52.487913 4754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.131@KRBTEST.COM'} at 127.4.151.131:41975
I20260619 08:18:52.495327 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 08:18:52 dist-test-slave-d3n8 krb5kdc[4709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857132, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 08:18:52 dist-test-slave-d3n8 krb5kdc[4709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857132, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.151.129@KRBTEST.COM
I20260619 08:18:52.519646 5220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:52.510905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37092 (local address 127.4.151.129:34719)
0619 08:18:52.511156 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:52.511169 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:52.511181 (+ 12us) server_negotiation.cc:407] Connection header received
0619 08:18:52.511208 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:52.511210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:52.511224 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:52.511300 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:52.511887 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:52.512694 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:52.513583 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:52.514126 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:52.514209 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:52.516755 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:52.516763 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:52.516763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:52.516771 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:52.518576 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:52.518932 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:52.518934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:52.518934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:52.518994 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:52.519155 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:52.519156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:52.519156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:52.519287 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:52.519404 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:52.519459 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 08:18:52.519564 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":184,"threads_started":1}
W20260619 08:18:52.520022 4859 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37092
Jun 19 08:18:52 dist-test-slave-d3n8 krb5kdc[4709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 08:18:52 dist-test-slave-d3n8 krb5kdc[4709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.129@KRBTEST.COM
I20260619 08:18:52.543464 5220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:52.535940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37102 (local address 127.4.151.129:34719)
0619 08:18:52.536001 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:52.536003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:52.536013 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:18:52.536046 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:52.536047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:52.536058 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:52.536122 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:52.536719 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:52.537283 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:52.538141 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:52.538356 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:52.538436 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:52.540861 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:52.540863 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:52.540863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:52.540871 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:52.542372 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:52.542727 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:52.542728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:52.542729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:52.542783 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:52.542972 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:52.542972 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:52.542973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:52.543071 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:52.543193 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:52.543275 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 08:18:52.543356 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 08:18:52.544036 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 4816
I20260619 08:18:52.549494 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 4950
I20260619 08:18:52.553969 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 5084
I20260619 08:18:52.558405 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 4722
2026-06-19T08:18:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1964 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:18:52 dist-test-slave-d3n8 krb5kdc[5227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:18:52 dist-test-slave-d3n8 krb5kdc[5227](info): set up 2 sockets
Jun 19 08:18:52 dist-test-slave-d3n8 krb5kdc[5227](info): commencing operation
krb5kdc: starting...
W20260619 08:18:53.612852 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:18:53 dist-test-slave-d3n8 krb5kdc[5227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:18:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:18:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:18:53.741286 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:33771
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:33771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:18:53.823621 5240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:18:53.823838 5240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:18:53.823863 5240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:18:53.825032 5240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:18:53.825078 5240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:18:53.825093 5240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:18:53.825105 5240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:18:53.825117 5240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:18:53.826707 5240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:33771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:33771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:18:53.826972 5240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:18:53.827203 5240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:18:53.832484 5246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:53.832536 5245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:18:53.832558 5240 server_base.cc:1034] running on GCE node
W20260619 08:18:53.832497 5248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:18:53.832898 5240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:18:53.833199 5240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:18:53.834309 5240 hybrid_clock.cc:648] HybridClock initialized: now 1781857133834308 us; error 44 us; skew 500 ppm
Jun 19 08:18:53 dist-test-slave-d3n8 krb5kdc[5227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857133, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:18:53.837627 5240 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:18:53.838160 5240 webserver.cc:460] Webserver started at http://127.4.151.190:39019/ using document root <none> and password file <none>
I20260619 08:18:53.838392 5240 fs_manager.cc:359] Metadata directory not provided
I20260619 08:18:53.838438 5240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:18:53.838574 5240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:18:53.839680 5240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "490d77ba5be842aebadee05d2a7fea8c"
format_stamp: "Formatted at 2026-06-19 08:18:53 on dist-test-slave-d3n8"
server_key: "a858025fef919f2b2f0ddb5ed0a8d473"
server_key_iv: "0085a7956199513b45ccbf8b15ebc78f"
server_key_version: "encryptionkey@0"
I20260619 08:18:53.840058 5240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "490d77ba5be842aebadee05d2a7fea8c"
format_stamp: "Formatted at 2026-06-19 08:18:53 on dist-test-slave-d3n8"
server_key: "a858025fef919f2b2f0ddb5ed0a8d473"
server_key_iv: "0085a7956199513b45ccbf8b15ebc78f"
server_key_version: "encryptionkey@0"
I20260619 08:18:53.841549 5240 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:18:53.842629 5255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:53.842980 5240 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:18:53.843109 5240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "490d77ba5be842aebadee05d2a7fea8c"
format_stamp: "Formatted at 2026-06-19 08:18:53 on dist-test-slave-d3n8"
server_key: "a858025fef919f2b2f0ddb5ed0a8d473"
server_key_iv: "0085a7956199513b45ccbf8b15ebc78f"
server_key_version: "encryptionkey@0"
I20260619 08:18:53.843194 5240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:18:53.852687 5240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:18:53.854954 5240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:18:53.855154 5240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:18:53.859210 5240 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:33771
I20260619 08:18:53.859274 5307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:33771 every 8 connection(s)
I20260619 08:18:53.859642 5240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:18:53.860666 5308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:18:53.863335 5308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 490d77ba5be842aebadee05d2a7fea8c: Bootstrap starting.
I20260619 08:18:53.864118 5308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 490d77ba5be842aebadee05d2a7fea8c: Neither blocks nor log segments found. Creating new log.
I20260619 08:18:53.864480 5308 log.cc:824] T 00000000000000000000000000000000 P 490d77ba5be842aebadee05d2a7fea8c: Log is configured to *not* fsync() on all Append() calls
I20260619 08:18:53.865302 5308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 490d77ba5be842aebadee05d2a7fea8c: No bootstrap required, opened a new log
I20260619 08:18:53.866626 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 5240
I20260619 08:18:53.866719 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:18:53.866882 4702 external_mini_cluster.cc:1428] Setting key 82722875c5bbb5010527f174fa82fe59
I20260619 08:18:53.867573 5308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 490d77ba5be842aebadee05d2a7fea8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490d77ba5be842aebadee05d2a7fea8c" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33771 } }
I20260619 08:18:53.867789 5308 raft_consensus.cc:374] T 00000000000000000000000000000000 P 490d77ba5be842aebadee05d2a7fea8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:18:53.867854 5308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 490d77ba5be842aebadee05d2a7fea8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 490d77ba5be842aebadee05d2a7fea8c, State: Initialized, Role: FOLLOWER
I20260619 08:18:53.868055 5308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 490d77ba5be842aebadee05d2a7fea8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490d77ba5be842aebadee05d2a7fea8c" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33771 } }
I20260619 08:18:53.868158 5308 raft_consensus.cc:388] T 00000000000000000000000000000000 P 490d77ba5be842aebadee05d2a7fea8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:18:53.868206 5308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 490d77ba5be842aebadee05d2a7fea8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:18:53.868271 5308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 490d77ba5be842aebadee05d2a7fea8c [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:18:53.869598 5308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 490d77ba5be842aebadee05d2a7fea8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490d77ba5be842aebadee05d2a7fea8c" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33771 } }
I20260619 08:18:53.869695 5308 leader_election.cc:302] T 00000000000000000000000000000000 P 490d77ba5be842aebadee05d2a7fea8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 490d77ba5be842aebadee05d2a7fea8c; no voters:
I20260619 08:18:53.869889 5308 leader_election.cc:288] T 00000000000000000000000000000000 P 490d77ba5be842aebadee05d2a7fea8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:18:53.869935 5313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 490d77ba5be842aebadee05d2a7fea8c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:18:53.870074 5313 raft_consensus.cc:686] T 00000000000000000000000000000000 P 490d77ba5be842aebadee05d2a7fea8c [term 1 LEADER]: Becoming Leader. State: Replica: 490d77ba5be842aebadee05d2a7fea8c, State: Running, Role: LEADER
I20260619 08:18:53.870182 5308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 490d77ba5be842aebadee05d2a7fea8c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:18:53.870163 5313 consensus_queue.cc:237] T 00000000000000000000000000000000 P 490d77ba5be842aebadee05d2a7fea8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490d77ba5be842aebadee05d2a7fea8c" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33771 } }
I20260619 08:18:53.870517 5313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 490d77ba5be842aebadee05d2a7fea8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 490d77ba5be842aebadee05d2a7fea8c. Latest consensus state: current_term: 1 leader_uuid: "490d77ba5be842aebadee05d2a7fea8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490d77ba5be842aebadee05d2a7fea8c" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33771 } } }
I20260619 08:18:53.870599 5313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 490d77ba5be842aebadee05d2a7fea8c [sys.catalog]: This master's current role is: LEADER
I20260619 08:18:53.870529 5315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 490d77ba5be842aebadee05d2a7fea8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "490d77ba5be842aebadee05d2a7fea8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490d77ba5be842aebadee05d2a7fea8c" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33771 } } }
I20260619 08:18:53.870656 5315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 490d77ba5be842aebadee05d2a7fea8c [sys.catalog]: This master's current role is: LEADER
W20260619 08:18:53.872213 5329 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 490d77ba5be842aebadee05d2a7fea8c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:18:53.872278 5329 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:18:53.872325 5318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:18:53.872478 5318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:18:53.874078 5318 catalog_manager.cc:1347] Generated new cluster ID: a6a0cf25327c453ca9c221eb52cc3d17
I20260619 08:18:53.874141 5318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:18:53 dist-test-slave-d3n8 krb5kdc[5227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:18:53.879364 5312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:53.867529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52530 (local address 127.4.151.190:33771)
0619 08:18:53.867752 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:53.867760 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:53.867790 (+ 30us) server_negotiation.cc:407] Connection header received
0619 08:18:53.868029 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:53.868036 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:53.868091 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:53.868248 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:53.869170 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:53.870497 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:53.871786 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:53.872500 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:53.872661 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:53.875573 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:53.875577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:53.875578 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:53.875590 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:53.877646 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:53.878031 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:53.878035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:53.878035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:53.878168 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:53.878364 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:53.878365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:53.878365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:53.878738 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:53.878941 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:53.879011 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 08:18:53.879150 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":109,"threads_started":1}
I20260619 08:18:53.897257 5318 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:18:53.897840 5318 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:18:53.903641 5318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 490d77ba5be842aebadee05d2a7fea8c: Generated new TSK 0
I20260619 08:18:53.903854 5318 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.151.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
I20260619 08:18:53.941058 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:33771
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:18:54.023752 5334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:18:54.024173 5334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:18:54.024221 5334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:18:54.025854 5334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:18:54.025918 5334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:18:54.025950 5334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:18:54.027577 5334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:33771
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:18:54.027859 5334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:18:54.028115 5334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:18:54.033339 5339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:54.033432 5342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:54.033452 5340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:18:54.033619 5334 server_base.cc:1034] running on GCE node
I20260619 08:18:54.033897 5334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:18:54.034173 5334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:18:54.035338 5334 hybrid_clock.cc:648] HybridClock initialized: now 1781857134035317 us; error 38 us; skew 500 ppm
Jun 19 08:18:54 dist-test-slave-d3n8 krb5kdc[5227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857134, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:18:54.038093 5334 init.cc:377] Logged in from keytab as kudu/127.4.151.129@KRBTEST.COM (short username kudu)
I20260619 08:18:54.038632 5334 webserver.cc:460] Webserver started at http://127.4.151.129:46757/ using document root <none> and password file <none>
I20260619 08:18:54.038870 5334 fs_manager.cc:359] Metadata directory not provided
I20260619 08:18:54.038918 5334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:18:54.039048 5334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:18:54.040227 5334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "212f7b1b35a443e78dec7b680d8353a0"
format_stamp: "Formatted at 2026-06-19 08:18:54 on dist-test-slave-d3n8"
server_key: "4e802a28f49569f692631d171a5c1f6d"
server_key_iv: "9ecbf805dd2595fefe29d9f060d4b62f"
server_key_version: "encryptionkey@0"
I20260619 08:18:54.040617 5334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "212f7b1b35a443e78dec7b680d8353a0"
format_stamp: "Formatted at 2026-06-19 08:18:54 on dist-test-slave-d3n8"
server_key: "4e802a28f49569f692631d171a5c1f6d"
server_key_iv: "9ecbf805dd2595fefe29d9f060d4b62f"
server_key_version: "encryptionkey@0"
I20260619 08:18:54.042187 5334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:18:54.043082 5349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:54.043278 5334 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:18:54.043388 5334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "212f7b1b35a443e78dec7b680d8353a0"
format_stamp: "Formatted at 2026-06-19 08:18:54 on dist-test-slave-d3n8"
server_key: "4e802a28f49569f692631d171a5c1f6d"
server_key_iv: "9ecbf805dd2595fefe29d9f060d4b62f"
server_key_version: "encryptionkey@0"
I20260619 08:18:54.043478 5334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:18:54.098677 5334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:18:54.100951 5334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:18:54.101118 5334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:18:54.101388 5334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:18:54.101718 5334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:18:54.101771 5334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:54.101814 5334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:18:54.101856 5334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:54.107757 5334 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:46365
I20260619 08:18:54.107811 5462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:46365 every 8 connection(s)
I20260619 08:18:54.108114 5334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
Jun 19 08:18:54 dist-test-slave-d3n8 krb5kdc[5227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857134, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:18:54.117241 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 5334
I20260619 08:18:54.117322 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:18:54.117524 4702 external_mini_cluster.cc:1428] Setting key 64aa0002debf43dcb849373d30763547
I20260619 08:18:54.118690 5312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:54.108844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:49847 (local address 127.4.151.190:33771)
0619 08:18:54.108970 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:54.108973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:54.109141 (+ 168us) server_negotiation.cc:407] Connection header received
0619 08:18:54.110248 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:54.110250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:54.110265 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:54.110353 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:54.111083 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.111622 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.112663 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.112862 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.112921 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:54.115175 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:54.115177 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:54.115177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:54.115187 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:54.116726 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:54.117463 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:54.117464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:54.117464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:54.117522 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:54.117911 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:54.117913 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:54.117913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:54.118056 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:54.118138 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:54.118443 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 08:18:54.118553 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 08:18:54.118703 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:54.108822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33771 (local address 127.4.151.129:49847)
0619 08:18:54.109004 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:54.109051 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:54.110105 (+ 1054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:54.110420 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:54.110424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:54.110773 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:18:54.110973 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:54.110978 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.111714 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.111715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:54.112575 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:54.112578 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.112985 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.112986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:54.113097 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:54.113634 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:18:54.113648 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:18:54.115087 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:18:54.116806 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:54.116814 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:54.116816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:54.117060 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:54.117629 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:54.117630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:54.117630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:54.117744 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:54.118139 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:18:54.118147 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:18:54.118297 (+ 150us) client_negotiation.cc:769] Sending connection context
0619 08:18:54.118384 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 08:18:54.118532 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260619 08:18:54.119210 5463 heartbeater.cc:347] Connected to a master server at 127.4.151.190:33771
I20260619 08:18:54.119329 5463 heartbeater.cc:464] Registering TS with master...
I20260619 08:18:54.119527 5463 heartbeater.cc:511] Master 127.4.151.190:33771 requested a full tablet report, sending...
I20260619 08:18:54.120091 5272 ts_manager.cc:194] Registered new tserver with Master: 212f7b1b35a443e78dec7b680d8353a0 (127.4.151.129:46365)
I20260619 08:18:54.120700 5272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.129@KRBTEST.COM'} at 127.4.151.129:49847
WARNING: no policy specified for kudu/127.4.151.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
I20260619 08:18:54.145197 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:33771
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:18:54.229473 5468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:18:54.229761 5468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:18:54.229808 5468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:18:54.231092 5468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:18:54.231165 5468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:18:54.231211 5468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:18:54.232746 5468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:33771
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:18:54.233019 5468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:18:54.233242 5468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:18:54.238526 5474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:54.238511 5473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:54.238516 5476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:18:54.238745 5468 server_base.cc:1034] running on GCE node
I20260619 08:18:54.238992 5468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:18:54.239223 5468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:18:54.240445 5468 hybrid_clock.cc:648] HybridClock initialized: now 1781857134240432 us; error 26 us; skew 500 ppm
Jun 19 08:18:54 dist-test-slave-d3n8 krb5kdc[5227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857134, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:18:54.243389 5468 init.cc:377] Logged in from keytab as kudu/127.4.151.130@KRBTEST.COM (short username kudu)
I20260619 08:18:54.243861 5468 webserver.cc:460] Webserver started at http://127.4.151.130:38473/ using document root <none> and password file <none>
I20260619 08:18:54.244104 5468 fs_manager.cc:359] Metadata directory not provided
I20260619 08:18:54.244153 5468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:18:54.244302 5468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:18:54.245571 5468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "c47c9887f507455a91efa46ac8ae5601"
format_stamp: "Formatted at 2026-06-19 08:18:54 on dist-test-slave-d3n8"
server_key: "8832039eed26e30b94c8804ff30e07c3"
server_key_iv: "2846dd72e32190cca98bee1cdea51a6a"
server_key_version: "encryptionkey@0"
I20260619 08:18:54.245987 5468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "c47c9887f507455a91efa46ac8ae5601"
format_stamp: "Formatted at 2026-06-19 08:18:54 on dist-test-slave-d3n8"
server_key: "8832039eed26e30b94c8804ff30e07c3"
server_key_iv: "2846dd72e32190cca98bee1cdea51a6a"
server_key_version: "encryptionkey@0"
I20260619 08:18:54.247653 5468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:18:54.248528 5483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:54.248731 5468 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:18:54.248829 5468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "c47c9887f507455a91efa46ac8ae5601"
format_stamp: "Formatted at 2026-06-19 08:18:54 on dist-test-slave-d3n8"
server_key: "8832039eed26e30b94c8804ff30e07c3"
server_key_iv: "2846dd72e32190cca98bee1cdea51a6a"
server_key_version: "encryptionkey@0"
I20260619 08:18:54.248909 5468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:18:54.269032 5468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:18:54.271589 5468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:18:54.271733 5468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:18:54.271998 5468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:18:54.272320 5468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:18:54.272369 5468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:54.272404 5468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:18:54.272439 5468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:54.278434 5468 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:39171
I20260619 08:18:54.278477 5596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:39171 every 8 connection(s)
I20260619 08:18:54.278841 5468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
I20260619 08:18:54.280519 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 5468
I20260619 08:18:54.280591 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:18:54.280759 4702 external_mini_cluster.cc:1428] Setting key a21829b4c70cc921bee2aa65d9242de9
Jun 19 08:18:54 dist-test-slave-d3n8 krb5kdc[5227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857134, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.151.131@KRBTEST.COM; defaulting to no policy
I20260619 08:18:54.289938 5312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:54.279649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:51505 (local address 127.4.151.190:33771)
0619 08:18:54.279715 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:54.279717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:54.279945 (+ 228us) server_negotiation.cc:407] Connection header received
0619 08:18:54.281179 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:54.281180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:54.281192 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:54.281247 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:54.281972 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.282492 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.283498 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.283718 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.283805 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:54.286116 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:54.286117 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:54.286118 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:54.286126 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:54.287817 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:54.288430 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:54.288431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:54.288432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:54.288499 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:54.288756 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:54.288757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:54.288758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:54.288909 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:54.289321 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:54.289722 (+ 401us) server_negotiation.cc:299] Negotiation successful
0619 08:18:54.289828 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 08:18:54.290122 5599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:54.279635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33771 (local address 127.4.151.130:51505)
0619 08:18:54.279822 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:54.279858 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:54.281077 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:54.281321 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:54.281325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:54.281686 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:18:54.281880 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:54.281884 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.282586 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.282587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:54.283398 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:54.283400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.283812 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.283813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:54.283947 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:54.284492 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:18:54.284501 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:18:54.286019 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:18:54.287937 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:54.287945 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:54.287948 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:54.288269 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:54.288577 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:54.288577 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:54.288578 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:54.288687 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:54.289306 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:18:54.289314 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:18:54.289552 (+ 238us) client_negotiation.cc:769] Sending connection context
0619 08:18:54.289703 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 08:18:54.289874 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
I20260619 08:18:54.290750 5597 heartbeater.cc:347] Connected to a master server at 127.4.151.190:33771
I20260619 08:18:54.290835 5597 heartbeater.cc:464] Registering TS with master...
I20260619 08:18:54.291116 5597 heartbeater.cc:511] Master 127.4.151.190:33771 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.131@KRBTEST.COM" created.
I20260619 08:18:54.291555 5272 ts_manager.cc:194] Registered new tserver with Master: c47c9887f507455a91efa46ac8ae5601 (127.4.151.130:39171)
I20260619 08:18:54.291962 5272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.130@KRBTEST.COM'} at 127.4.151.130:51505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
I20260619 08:18:54.309556 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:33771
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:18:54.394431 5602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:18:54.394654 5602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:18:54.394680 5602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:18:54.395833 5602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:18:54.395872 5602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:18:54.395900 5602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:18:54.397429 5602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:33771
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:18:54.397636 5602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:18:54.397830 5602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:18:54.403045 5607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:18:54.403151 5602 server_base.cc:1034] running on GCE node
W20260619 08:18:54.403203 5610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:54.403115 5608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:18:54.403422 5602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:18:54.403707 5602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:18:54.404892 5602 hybrid_clock.cc:648] HybridClock initialized: now 1781857134404839 us; error 70 us; skew 500 ppm
Jun 19 08:18:54 dist-test-slave-d3n8 krb5kdc[5227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857134, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:18:54.407747 5602 init.cc:377] Logged in from keytab as kudu/127.4.151.131@KRBTEST.COM (short username kudu)
I20260619 08:18:54.408224 5602 webserver.cc:460] Webserver started at http://127.4.151.131:35565/ using document root <none> and password file <none>
I20260619 08:18:54.408478 5602 fs_manager.cc:359] Metadata directory not provided
I20260619 08:18:54.408545 5602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:18:54.408687 5602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:18:54.409852 5602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "0bc1c5da683a4a0e962d5c2bdacc6e25"
format_stamp: "Formatted at 2026-06-19 08:18:54 on dist-test-slave-d3n8"
server_key: "c7d6090b925dc4be309bd899e46a0c82"
server_key_iv: "1563bc990f88037e1679ec5bdd4143e4"
server_key_version: "encryptionkey@0"
I20260619 08:18:54.410211 5602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "0bc1c5da683a4a0e962d5c2bdacc6e25"
format_stamp: "Formatted at 2026-06-19 08:18:54 on dist-test-slave-d3n8"
server_key: "c7d6090b925dc4be309bd899e46a0c82"
server_key_iv: "1563bc990f88037e1679ec5bdd4143e4"
server_key_version: "encryptionkey@0"
I20260619 08:18:54.411624 5602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:18:54.412674 5617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:54.412984 5602 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:18:54.413100 5602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "0bc1c5da683a4a0e962d5c2bdacc6e25"
format_stamp: "Formatted at 2026-06-19 08:18:54 on dist-test-slave-d3n8"
server_key: "c7d6090b925dc4be309bd899e46a0c82"
server_key_iv: "1563bc990f88037e1679ec5bdd4143e4"
server_key_version: "encryptionkey@0"
I20260619 08:18:54.413177 5602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:18:54.427388 5602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:18:54.429898 5602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:18:54.430066 5602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:18:54.430301 5602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:18:54.430612 5602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:18:54.430661 5602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:54.430704 5602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:18:54.430730 5602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:54.436619 5602 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:38587
I20260619 08:18:54.436677 5730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:38587 every 8 connection(s)
I20260619 08:18:54.437026 5602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
Jun 19 08:18:54 dist-test-slave-d3n8 krb5kdc[5227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857134, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:18:54.444729 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 5602
I20260619 08:18:54.444839 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:18:54.445098 4702 external_mini_cluster.cc:1428] Setting key edfc2321b877ee941ab1f2b3ce4026a8
I20260619 08:18:54.447850 5312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:54.437838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:36191 (local address 127.4.151.190:33771)
0619 08:18:54.437916 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:54.437919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:54.438047 (+ 128us) server_negotiation.cc:407] Connection header received
0619 08:18:54.439152 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:54.439154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:54.439170 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:54.439241 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:54.440012 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.440537 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.441708 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.441978 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.442053 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:54.444315 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:54.444317 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:54.444317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:54.444324 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:54.446273 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:54.446682 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:54.446683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:54.446683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:54.446744 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:54.447254 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:54.447255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:54.447255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:54.447380 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:54.447446 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:54.447718 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 08:18:54.447785 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 08:18:54.448025 5733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:54.437764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33771 (local address 127.4.151.131:36191)
0619 08:18:54.437912 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:54.437952 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:54.438962 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:54.439312 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:54.439316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:54.439618 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:18:54.439835 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:54.439840 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.440635 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.440636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:54.441610 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:54.441613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.442085 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.442087 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:54.442205 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:54.442746 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:18:54.442755 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:18:54.444216 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:18:54.446357 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:54.446363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:54.446365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:54.446617 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:54.447036 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:54.447037 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:54.447037 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:54.447162 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:54.447466 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:18:54.447470 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:18:54.447600 (+ 130us) client_negotiation.cc:769] Sending connection context
0619 08:18:54.447689 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 08:18:54.447818 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260619 08:18:54.448503 5731 heartbeater.cc:347] Connected to a master server at 127.4.151.190:33771
I20260619 08:18:54.448597 5731 heartbeater.cc:464] Registering TS with master...
I20260619 08:18:54.448747 5731 heartbeater.cc:511] Master 127.4.151.190:33771 requested a full tablet report, sending...
I20260619 08:18:54.449098 5272 ts_manager.cc:194] Registered new tserver with Master: 0bc1c5da683a4a0e962d5c2bdacc6e25 (127.4.151.131:38587)
I20260619 08:18:54.449507 5272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.131@KRBTEST.COM'} at 127.4.151.131:36191
I20260619 08:18:54.457333 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:18:54.465734 5312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:54.459359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52534 (local address 127.4.151.190:33771)
0619 08:18:54.459430 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:54.459433 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:54.459513 (+ 80us) server_negotiation.cc:407] Connection header received
0619 08:18:54.459602 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:54.459603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:54.459613 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:54.459671 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:54.460279 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.460798 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.461583 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.461786 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.461868 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:54.463124 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:54.463125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:54.463125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:54.463132 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:54.464546 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:54.464892 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:54.464893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:54.464894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:54.464955 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:54.465145 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:54.465146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:54.465146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:54.465304 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:54.465447 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:54.465549 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 08:18:54.465631 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 08:18:54.468008 5272 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:18:54.469638 5272 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:18:54.476836 5745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:54.472433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36622 (local address 127.4.151.131:38587)
0619 08:18:54.472739 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:54.472743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:54.472751 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:18:54.472801 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:54.472804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:54.472821 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:54.472954 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:18:54.473192 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.474344 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.475577 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.476350 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.476429 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:54.476489 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:54.476716 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 08:18:54.476757 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":88,"threads_started":1}
I20260619 08:18:54.477123 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:54.472434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60846 (local address 127.4.151.130:39171)
0619 08:18:54.473099 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:54.473103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:54.473112 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:18:54.473147 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:54.473150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:54.473164 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:54.473237 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:18:54.473367 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.474189 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.475754 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.476461 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.476526 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:54.476581 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:54.476972 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 08:18:54.477019 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":80,"threads_started":1}
I20260619 08:18:54.476861 5742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:54.472247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:39171 (local address 127.0.0.1:60846)
0619 08:18:54.472350 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:54.472389 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:54.472448 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:54.473234 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:54.473235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:54.473238 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:18:54.473272 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:54.473273 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.474320 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.474321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:54.475659 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:54.475661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.476542 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.476543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:54.476599 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:54.476602 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:18:54.476687 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 08:18:54.476753 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":63,"thread_start_us":58,"threads_started":1}
I20260619 08:18:54.477790 5746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:54.472445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:46365 (local address 127.0.0.1:39498)
0619 08:18:54.472680 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:54.472690 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:54.472739 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:54.473109 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:54.473110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:54.473112 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:18:54.473168 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:54.473170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.474523 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.474524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:54.475659 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:54.475661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.477507 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.477508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:54.477590 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:54.477595 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:18:54.477656 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 08:18:54.477721 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":26,"thread_start_us":81,"threads_started":1}
I20260619 08:18:54.477803 5747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:54.472678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39498 (local address 127.4.151.129:46365)
0619 08:18:54.472859 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:54.472864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:54.472891 (+ 27us) server_negotiation.cc:407] Connection header received
0619 08:18:54.472929 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:54.472931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:54.472945 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:54.473016 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:18:54.473341 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.474423 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.476611 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.477327 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.477420 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:54.477474 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:54.477687 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 08:18:54.477732 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
I20260619 08:18:54.476861 5743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:54.472286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:38587 (local address 127.0.0.1:36622)
0619 08:18:54.472593 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:54.472603 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:54.472655 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:54.473027 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:54.473030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:54.473032 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:18:54.473105 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:54.473107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.474459 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.474460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:54.475474 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:54.475476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.476453 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.476454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:54.476580 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:54.476598 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 08:18:54.476679 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 08:18:54.476752 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":87,"threads_started":1}
I20260619 08:18:54.478276 5531 tablet_service.cc:1467] Processing CreateTablet for tablet 8b20598c4f70472f8ff987eff5a20fe5 (DEFAULT_TABLE table=test-table [id=af89b968865d4d66925fbda379e00027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:18:54.478602 5531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b20598c4f70472f8ff987eff5a20fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:18:54.478705 5397 tablet_service.cc:1467] Processing CreateTablet for tablet 8b20598c4f70472f8ff987eff5a20fe5 (DEFAULT_TABLE table=test-table [id=af89b968865d4d66925fbda379e00027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:18:54.478710 5665 tablet_service.cc:1467] Processing CreateTablet for tablet 8b20598c4f70472f8ff987eff5a20fe5 (DEFAULT_TABLE table=test-table [id=af89b968865d4d66925fbda379e00027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:18:54.479055 5665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b20598c4f70472f8ff987eff5a20fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:18:54.479142 5397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b20598c4f70472f8ff987eff5a20fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:18:54.481096 5748 tablet_bootstrap.cc:492] T 8b20598c4f70472f8ff987eff5a20fe5 P c47c9887f507455a91efa46ac8ae5601: Bootstrap starting.
I20260619 08:18:54.481766 5748 tablet_bootstrap.cc:654] T 8b20598c4f70472f8ff987eff5a20fe5 P c47c9887f507455a91efa46ac8ae5601: Neither blocks nor log segments found. Creating new log.
I20260619 08:18:54.482261 5748 log.cc:824] T 8b20598c4f70472f8ff987eff5a20fe5 P c47c9887f507455a91efa46ac8ae5601: Log is configured to *not* fsync() on all Append() calls
I20260619 08:18:54.482625 5749 tablet_bootstrap.cc:492] T 8b20598c4f70472f8ff987eff5a20fe5 P 212f7b1b35a443e78dec7b680d8353a0: Bootstrap starting.
I20260619 08:18:54.482939 5751 tablet_bootstrap.cc:492] T 8b20598c4f70472f8ff987eff5a20fe5 P 0bc1c5da683a4a0e962d5c2bdacc6e25: Bootstrap starting.
I20260619 08:18:54.483266 5748 tablet_bootstrap.cc:492] T 8b20598c4f70472f8ff987eff5a20fe5 P c47c9887f507455a91efa46ac8ae5601: No bootstrap required, opened a new log
I20260619 08:18:54.483338 5748 ts_tablet_manager.cc:1397] T 8b20598c4f70472f8ff987eff5a20fe5 P c47c9887f507455a91efa46ac8ae5601: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 08:18:54.483543 5749 tablet_bootstrap.cc:654] T 8b20598c4f70472f8ff987eff5a20fe5 P 212f7b1b35a443e78dec7b680d8353a0: Neither blocks nor log segments found. Creating new log.
I20260619 08:18:54.483597 5751 tablet_bootstrap.cc:654] T 8b20598c4f70472f8ff987eff5a20fe5 P 0bc1c5da683a4a0e962d5c2bdacc6e25: Neither blocks nor log segments found. Creating new log.
I20260619 08:18:54.484087 5751 log.cc:824] T 8b20598c4f70472f8ff987eff5a20fe5 P 0bc1c5da683a4a0e962d5c2bdacc6e25: Log is configured to *not* fsync() on all Append() calls
I20260619 08:18:54.484086 5749 log.cc:824] T 8b20598c4f70472f8ff987eff5a20fe5 P 212f7b1b35a443e78dec7b680d8353a0: Log is configured to *not* fsync() on all Append() calls
I20260619 08:18:54.484871 5751 tablet_bootstrap.cc:492] T 8b20598c4f70472f8ff987eff5a20fe5 P 0bc1c5da683a4a0e962d5c2bdacc6e25: No bootstrap required, opened a new log
I20260619 08:18:54.484951 5751 ts_tablet_manager.cc:1397] T 8b20598c4f70472f8ff987eff5a20fe5 P 0bc1c5da683a4a0e962d5c2bdacc6e25: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:18:54.485167 5749 tablet_bootstrap.cc:492] T 8b20598c4f70472f8ff987eff5a20fe5 P 212f7b1b35a443e78dec7b680d8353a0: No bootstrap required, opened a new log
I20260619 08:18:54.485276 5749 ts_tablet_manager.cc:1397] T 8b20598c4f70472f8ff987eff5a20fe5 P 212f7b1b35a443e78dec7b680d8353a0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 08:18:54.485263 5748 raft_consensus.cc:348] T 8b20598c4f70472f8ff987eff5a20fe5 P c47c9887f507455a91efa46ac8ae5601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47c9887f507455a91efa46ac8ae5601" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39171 } } peers { permanent_uuid: "212f7b1b35a443e78dec7b680d8353a0" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 46365 } } peers { permanent_uuid: "0bc1c5da683a4a0e962d5c2bdacc6e25" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 38587 } }
I20260619 08:18:54.485476 5748 raft_consensus.cc:374] T 8b20598c4f70472f8ff987eff5a20fe5 P c47c9887f507455a91efa46ac8ae5601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:18:54.485522 5748 raft_consensus.cc:729] T 8b20598c4f70472f8ff987eff5a20fe5 P c47c9887f507455a91efa46ac8ae5601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c47c9887f507455a91efa46ac8ae5601, State: Initialized, Role: FOLLOWER
I20260619 08:18:54.485641 5748 consensus_queue.cc:260] T 8b20598c4f70472f8ff987eff5a20fe5 P c47c9887f507455a91efa46ac8ae5601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47c9887f507455a91efa46ac8ae5601" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39171 } } peers { permanent_uuid: "212f7b1b35a443e78dec7b680d8353a0" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 46365 } } peers { permanent_uuid: "0bc1c5da683a4a0e962d5c2bdacc6e25" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 38587 } }
I20260619 08:18:54.485924 5748 ts_tablet_manager.cc:1428] T 8b20598c4f70472f8ff987eff5a20fe5 P c47c9887f507455a91efa46ac8ae5601: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260619 08:18:54.486053 5597 heartbeater.cc:503] Master 127.4.151.190:33771 was elected leader, sending a full tablet report...
I20260619 08:18:54.487133 5751 raft_consensus.cc:348] T 8b20598c4f70472f8ff987eff5a20fe5 P 0bc1c5da683a4a0e962d5c2bdacc6e25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47c9887f507455a91efa46ac8ae5601" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39171 } } peers { permanent_uuid: "212f7b1b35a443e78dec7b680d8353a0" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 46365 } } peers { permanent_uuid: "0bc1c5da683a4a0e962d5c2bdacc6e25" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 38587 } }
I20260619 08:18:54.487326 5751 raft_consensus.cc:374] T 8b20598c4f70472f8ff987eff5a20fe5 P 0bc1c5da683a4a0e962d5c2bdacc6e25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:18:54.487365 5751 raft_consensus.cc:729] T 8b20598c4f70472f8ff987eff5a20fe5 P 0bc1c5da683a4a0e962d5c2bdacc6e25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc1c5da683a4a0e962d5c2bdacc6e25, State: Initialized, Role: FOLLOWER
I20260619 08:18:54.487502 5751 consensus_queue.cc:260] T 8b20598c4f70472f8ff987eff5a20fe5 P 0bc1c5da683a4a0e962d5c2bdacc6e25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47c9887f507455a91efa46ac8ae5601" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39171 } } peers { permanent_uuid: "212f7b1b35a443e78dec7b680d8353a0" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 46365 } } peers { permanent_uuid: "0bc1c5da683a4a0e962d5c2bdacc6e25" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 38587 } }
I20260619 08:18:54.487504 5749 raft_consensus.cc:348] T 8b20598c4f70472f8ff987eff5a20fe5 P 212f7b1b35a443e78dec7b680d8353a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47c9887f507455a91efa46ac8ae5601" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39171 } } peers { permanent_uuid: "212f7b1b35a443e78dec7b680d8353a0" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 46365 } } peers { permanent_uuid: "0bc1c5da683a4a0e962d5c2bdacc6e25" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 38587 } }
I20260619 08:18:54.487677 5749 raft_consensus.cc:374] T 8b20598c4f70472f8ff987eff5a20fe5 P 212f7b1b35a443e78dec7b680d8353a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:18:54.487727 5749 raft_consensus.cc:729] T 8b20598c4f70472f8ff987eff5a20fe5 P 212f7b1b35a443e78dec7b680d8353a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212f7b1b35a443e78dec7b680d8353a0, State: Initialized, Role: FOLLOWER
I20260619 08:18:54.487809 5731 heartbeater.cc:503] Master 127.4.151.190:33771 was elected leader, sending a full tablet report...
I20260619 08:18:54.487886 5749 consensus_queue.cc:260] T 8b20598c4f70472f8ff987eff5a20fe5 P 212f7b1b35a443e78dec7b680d8353a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47c9887f507455a91efa46ac8ae5601" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39171 } } peers { permanent_uuid: "212f7b1b35a443e78dec7b680d8353a0" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 46365 } } peers { permanent_uuid: "0bc1c5da683a4a0e962d5c2bdacc6e25" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 38587 } }
I20260619 08:18:54.488186 5749 ts_tablet_manager.cc:1428] T 8b20598c4f70472f8ff987eff5a20fe5 P 212f7b1b35a443e78dec7b680d8353a0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:18:54.487799 5751 ts_tablet_manager.cc:1428] T 8b20598c4f70472f8ff987eff5a20fe5 P 0bc1c5da683a4a0e962d5c2bdacc6e25: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:18:54.488183 5463 heartbeater.cc:503] Master 127.4.151.190:33771 was elected leader, sending a full tablet report...
W20260619 08:18:54.529491 5598 tablet.cc:2369] T 8b20598c4f70472f8ff987eff5a20fe5 P c47c9887f507455a91efa46ac8ae5601: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:18:54.608856 5464 tablet.cc:2369] T 8b20598c4f70472f8ff987eff5a20fe5 P 212f7b1b35a443e78dec7b680d8353a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:18:54.609458 5756 raft_consensus.cc:482] T 8b20598c4f70472f8ff987eff5a20fe5 P 212f7b1b35a443e78dec7b680d8353a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:18:54.609557 5756 raft_consensus.cc:504] T 8b20598c4f70472f8ff987eff5a20fe5 P 212f7b1b35a443e78dec7b680d8353a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47c9887f507455a91efa46ac8ae5601" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39171 } } peers { permanent_uuid: "212f7b1b35a443e78dec7b680d8353a0" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 46365 } } peers { permanent_uuid: "0bc1c5da683a4a0e962d5c2bdacc6e25" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 38587 } }
I20260619 08:18:54.610003 5756 leader_election.cc:288] T 8b20598c4f70472f8ff987eff5a20fe5 P 212f7b1b35a443e78dec7b680d8353a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c47c9887f507455a91efa46ac8ae5601 (127.4.151.130:39171), 0bc1c5da683a4a0e962d5c2bdacc6e25 (127.4.151.131:38587)
I20260619 08:18:54.612900 5745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:54.610333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:44213 (local address 127.4.151.131:38587)
0619 08:18:54.610407 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:54.610409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:54.610418 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:18:54.610455 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:54.610456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:54.610463 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:54.610522 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:18:54.610704 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.611215 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.612170 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.612549 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.612633 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:54.612642 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:54.612791 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 08:18:54.612828 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 08:18:54.612905 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:54.610235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:38587 (local address 127.4.151.129:44213)
0619 08:18:54.610298 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:54.610310 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:54.610354 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:54.610542 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:54.610543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:54.610545 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:18:54.610631 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:54.610633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.611305 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.611306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:54.612076 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:54.612078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.612640 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.612641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:54.612717 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:54.612720 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:18:54.612783 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 08:18:54.612827 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 08:18:54.612979 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:54.610325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:55629 (local address 127.4.151.130:39171)
0619 08:18:54.610383 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:54.610386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:54.610500 (+ 114us) server_negotiation.cc:407] Connection header received
0619 08:18:54.610573 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:54.610574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:54.610586 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:54.610632 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:18:54.610801 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.611262 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.612219 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.612610 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.612699 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:54.612707 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:54.612875 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 08:18:54.612912 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 08:18:54.613209 5757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:54.610235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:39171 (local address 127.4.151.129:55629)
0619 08:18:54.610411 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:54.610421 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:54.610517 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:54.610648 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:54.610649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:54.610650 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:18:54.610726 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:54.610727 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.611349 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.611350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:54.612099 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:54.612101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.612705 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.612706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:54.612786 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:54.612789 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:18:54.612867 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 08:18:54.612908 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":23,"thread_start_us":96,"threads_started":1}
I20260619 08:18:54.613255 5685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b20598c4f70472f8ff987eff5a20fe5" candidate_uuid: "212f7b1b35a443e78dec7b680d8353a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc1c5da683a4a0e962d5c2bdacc6e25" is_pre_election: true
I20260619 08:18:54.613446 5685 raft_consensus.cc:2454] T 8b20598c4f70472f8ff987eff5a20fe5 P 0bc1c5da683a4a0e962d5c2bdacc6e25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212f7b1b35a443e78dec7b680d8353a0 in term 0.
I20260619 08:18:54.613552 5551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b20598c4f70472f8ff987eff5a20fe5" candidate_uuid: "212f7b1b35a443e78dec7b680d8353a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c47c9887f507455a91efa46ac8ae5601" is_pre_election: true
I20260619 08:18:54.613687 5551 raft_consensus.cc:2454] T 8b20598c4f70472f8ff987eff5a20fe5 P c47c9887f507455a91efa46ac8ae5601 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212f7b1b35a443e78dec7b680d8353a0 in term 0.
I20260619 08:18:54.613750 5350 leader_election.cc:302] T 8b20598c4f70472f8ff987eff5a20fe5 P 212f7b1b35a443e78dec7b680d8353a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc1c5da683a4a0e962d5c2bdacc6e25, 212f7b1b35a443e78dec7b680d8353a0; no voters:
I20260619 08:18:54.613911 5756 raft_consensus.cc:2781] T 8b20598c4f70472f8ff987eff5a20fe5 P 212f7b1b35a443e78dec7b680d8353a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:18:54.613978 5756 raft_consensus.cc:482] T 8b20598c4f70472f8ff987eff5a20fe5 P 212f7b1b35a443e78dec7b680d8353a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:18:54.614003 5756 raft_consensus.cc:3037] T 8b20598c4f70472f8ff987eff5a20fe5 P 212f7b1b35a443e78dec7b680d8353a0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:18:54.614677 5756 raft_consensus.cc:504] T 8b20598c4f70472f8ff987eff5a20fe5 P 212f7b1b35a443e78dec7b680d8353a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47c9887f507455a91efa46ac8ae5601" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39171 } } peers { permanent_uuid: "212f7b1b35a443e78dec7b680d8353a0" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 46365 } } peers { permanent_uuid: "0bc1c5da683a4a0e962d5c2bdacc6e25" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 38587 } }
I20260619 08:18:54.614826 5756 leader_election.cc:288] T 8b20598c4f70472f8ff987eff5a20fe5 P 212f7b1b35a443e78dec7b680d8353a0 [CANDIDATE]: Term 1 election: Requested vote from peers c47c9887f507455a91efa46ac8ae5601 (127.4.151.130:39171), 0bc1c5da683a4a0e962d5c2bdacc6e25 (127.4.151.131:38587)
I20260619 08:18:54.615043 5685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b20598c4f70472f8ff987eff5a20fe5" candidate_uuid: "212f7b1b35a443e78dec7b680d8353a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc1c5da683a4a0e962d5c2bdacc6e25"
I20260619 08:18:54.615043 5551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b20598c4f70472f8ff987eff5a20fe5" candidate_uuid: "212f7b1b35a443e78dec7b680d8353a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c47c9887f507455a91efa46ac8ae5601"
I20260619 08:18:54.615124 5551 raft_consensus.cc:3037] T 8b20598c4f70472f8ff987eff5a20fe5 P c47c9887f507455a91efa46ac8ae5601 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:18:54.615124 5685 raft_consensus.cc:3037] T 8b20598c4f70472f8ff987eff5a20fe5 P 0bc1c5da683a4a0e962d5c2bdacc6e25 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:18:54.616006 5685 raft_consensus.cc:2454] T 8b20598c4f70472f8ff987eff5a20fe5 P 0bc1c5da683a4a0e962d5c2bdacc6e25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212f7b1b35a443e78dec7b680d8353a0 in term 1.
I20260619 08:18:54.616045 5551 raft_consensus.cc:2454] T 8b20598c4f70472f8ff987eff5a20fe5 P c47c9887f507455a91efa46ac8ae5601 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212f7b1b35a443e78dec7b680d8353a0 in term 1.
I20260619 08:18:54.616233 5353 leader_election.cc:302] T 8b20598c4f70472f8ff987eff5a20fe5 P 212f7b1b35a443e78dec7b680d8353a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212f7b1b35a443e78dec7b680d8353a0, c47c9887f507455a91efa46ac8ae5601; no voters:
I20260619 08:18:54.616384 5756 raft_consensus.cc:2781] T 8b20598c4f70472f8ff987eff5a20fe5 P 212f7b1b35a443e78dec7b680d8353a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:18:54.616578 5756 raft_consensus.cc:686] T 8b20598c4f70472f8ff987eff5a20fe5 P 212f7b1b35a443e78dec7b680d8353a0 [term 1 LEADER]: Becoming Leader. State: Replica: 212f7b1b35a443e78dec7b680d8353a0, State: Running, Role: LEADER
I20260619 08:18:54.616685 5756 consensus_queue.cc:237] T 8b20598c4f70472f8ff987eff5a20fe5 P 212f7b1b35a443e78dec7b680d8353a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47c9887f507455a91efa46ac8ae5601" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39171 } } peers { permanent_uuid: "212f7b1b35a443e78dec7b680d8353a0" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 46365 } } peers { permanent_uuid: "0bc1c5da683a4a0e962d5c2bdacc6e25" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 38587 } }
I20260619 08:18:54.617611 5271 catalog_manager.cc:5495] T 8b20598c4f70472f8ff987eff5a20fe5 P 212f7b1b35a443e78dec7b680d8353a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 212f7b1b35a443e78dec7b680d8353a0 (127.4.151.129). New cstate: current_term: 1 leader_uuid: "212f7b1b35a443e78dec7b680d8353a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47c9887f507455a91efa46ac8ae5601" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212f7b1b35a443e78dec7b680d8353a0" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bc1c5da683a4a0e962d5c2bdacc6e25" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 38587 } health_report { overall_health: UNKNOWN } } }
Jun 19 08:18:54 dist-test-slave-d3n8 krb5kdc[5227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.129@KRBTEST.COM
I20260619 08:18:54.663189 5747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:54.655106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39506 (local address 127.4.151.129:46365)
0619 08:18:54.655179 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:54.655181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:54.655270 (+ 89us) server_negotiation.cc:407] Connection header received
0619 08:18:54.655362 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:54.655363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:54.655377 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:54.655428 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:54.656031 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.656520 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.657230 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.657533 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.657605 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:54.660161 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:54.660173 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:54.660174 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:54.660181 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:54.661971 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:54.662359 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:54.662362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:54.662363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:54.662444 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:54.662673 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:54.662674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:54.662674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:54.662833 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:54.662989 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:54.663025 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 08:18:54.663102 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 08:18:54 dist-test-slave-d3n8 krb5kdc[5227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857134, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 08:18:54 dist-test-slave-d3n8 krb5kdc[5227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857134, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.151.129@KRBTEST.COM
I20260619 08:18:54.686462 5747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:54.678953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39518 (local address 127.4.151.129:46365)
0619 08:18:54.679016 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:54.679018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:54.679210 (+ 192us) server_negotiation.cc:407] Connection header received
0619 08:18:54.679245 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:54.679245 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:54.679256 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:54.679309 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:54.679946 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.680428 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.681275 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.681620 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.681677 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:54.684199 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:54.684201 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:54.684201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:54.684207 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:54.685502 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:54.685815 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:54.685815 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:54.685816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:54.685863 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:54.686029 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:54.686029 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:54.686030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:54.686115 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:54.686185 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:54.686335 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 08:18:54.686396 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260619 08:18:54.687722 5732 tablet.cc:2369] T 8b20598c4f70472f8ff987eff5a20fe5 P 0bc1c5da683a4a0e962d5c2bdacc6e25: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 08:18:54 dist-test-slave-d3n8 krb5kdc[5227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 08:18:54 dist-test-slave-d3n8 krb5kdc[5227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.129@KRBTEST.COM
I20260619 08:18:54.708959 5747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:54.701562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39526 (local address 127.4.151.129:46365)
0619 08:18:54.701662 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:54.701664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:54.701756 (+ 92us) server_negotiation.cc:407] Connection header received
0619 08:18:54.701854 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:54.701855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:54.701865 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:54.701934 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:54.702575 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.703050 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.703772 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:54.703963 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:54.704038 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:54.706378 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:54.706380 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:54.706381 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:54.706389 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:54.708084 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:54.708401 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:54.708402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:54.708402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:54.708453 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:54.708590 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:54.708590 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:54.708590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:54.708681 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:54.708756 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:54.708839 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 08:18:54.708893 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 08:18:54.709522 5377 tablet_service.cc:2908] Rejecting scan request for tablet 8b20598c4f70472f8ff987eff5a20fe5: Uninitialized: clean time has not yet been initialized
I20260619 08:18:54.710546 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 5334
I20260619 08:18:54.716313 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 5468
I20260619 08:18:54.720942 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 5602
I20260619 08:18:54.725358 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 5240
2026-06-19T08:18:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2167 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:18:54 dist-test-slave-d3n8 krb5kdc[5772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:18:54 dist-test-slave-d3n8 krb5kdc[5772](info): set up 2 sockets
Jun 19 08:18:54 dist-test-slave-d3n8 krb5kdc[5772](info): commencing operation
krb5kdc: starting...
W20260619 08:18:55.825032 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:18:55 dist-test-slave-d3n8 krb5kdc[5772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:18:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:18:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:18:55.953157 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:45811
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:45811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:18:56.036252 5785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:18:56.036507 5785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:18:56.036547 5785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:18:56.037916 5785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:18:56.037984 5785 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 08:18:56.038005 5785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:18:56.038028 5785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:18:56.038057 5785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:18:56.038093 5785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:18:56.039630 5785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:45811
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:45811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:18:56.039925 5785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:18:56.040162 5785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:18:56.045548 5791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:56.045548 5790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:18:56.045603 5785 server_base.cc:1034] running on GCE node
W20260619 08:18:56.045611 5793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:18:56.046049 5785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:18:56.046375 5785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:18:56.047545 5785 hybrid_clock.cc:648] HybridClock initialized: now 1781857136047530 us; error 29 us; skew 500 ppm
Jun 19 08:18:56 dist-test-slave-d3n8 krb5kdc[5772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857136, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:18:56.057126 5785 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:18:56.057713 5785 webserver.cc:460] Webserver started at http://127.4.151.190:40173/ using document root <none> and password file <none>
I20260619 08:18:56.057997 5785 fs_manager.cc:359] Metadata directory not provided
I20260619 08:18:56.058058 5785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:18:56.058188 5785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:18:56.059315 5785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "50cfc77216e3413a8673d9aff3cce300"
format_stamp: "Formatted at 2026-06-19 08:18:56 on dist-test-slave-d3n8"
server_key: "0bb20b6dcc17b72163c46ef9a296993a"
server_key_iv: "652e6cee94f6b518c95dd4f61aa74a6a"
server_key_version: "encryptionkey@0"
I20260619 08:18:56.059682 5785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "50cfc77216e3413a8673d9aff3cce300"
format_stamp: "Formatted at 2026-06-19 08:18:56 on dist-test-slave-d3n8"
server_key: "0bb20b6dcc17b72163c46ef9a296993a"
server_key_iv: "652e6cee94f6b518c95dd4f61aa74a6a"
server_key_version: "encryptionkey@0"
I20260619 08:18:56.061096 5785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:18:56.061916 5800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:56.062129 5785 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 08:18:56.062222 5785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "50cfc77216e3413a8673d9aff3cce300"
format_stamp: "Formatted at 2026-06-19 08:18:56 on dist-test-slave-d3n8"
server_key: "0bb20b6dcc17b72163c46ef9a296993a"
server_key_iv: "652e6cee94f6b518c95dd4f61aa74a6a"
server_key_version: "encryptionkey@0"
I20260619 08:18:56.062278 5785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:18:56.079231 5785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:18:56.081584 5785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:18:56.081712 5785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:18:56.086493 5785 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:45811
I20260619 08:18:56.086572 5862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:45811 every 8 connection(s)
I20260619 08:18:56.086921 5785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:18:56.087647 5863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:18:56.088408 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 5785
I20260619 08:18:56.088516 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:18:56.088683 4702 external_mini_cluster.cc:1428] Setting key 21982147e63d9d0b49ee44d388bcb310
I20260619 08:18:56.090754 5863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50cfc77216e3413a8673d9aff3cce300: Bootstrap starting.
I20260619 08:18:56.091467 5863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50cfc77216e3413a8673d9aff3cce300: Neither blocks nor log segments found. Creating new log.
I20260619 08:18:56.091782 5863 log.cc:824] T 00000000000000000000000000000000 P 50cfc77216e3413a8673d9aff3cce300: Log is configured to *not* fsync() on all Append() calls
I20260619 08:18:56.092582 5863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50cfc77216e3413a8673d9aff3cce300: No bootstrap required, opened a new log
I20260619 08:18:56.094604 5863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50cfc77216e3413a8673d9aff3cce300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cfc77216e3413a8673d9aff3cce300" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 45811 } }
I20260619 08:18:56.094789 5863 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50cfc77216e3413a8673d9aff3cce300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:18:56.094825 5863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50cfc77216e3413a8673d9aff3cce300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50cfc77216e3413a8673d9aff3cce300, State: Initialized, Role: FOLLOWER
I20260619 08:18:56.094966 5863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50cfc77216e3413a8673d9aff3cce300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cfc77216e3413a8673d9aff3cce300" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 45811 } }
I20260619 08:18:56.095070 5863 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50cfc77216e3413a8673d9aff3cce300 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:18:56.095103 5863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50cfc77216e3413a8673d9aff3cce300 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:18:56.095150 5863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50cfc77216e3413a8673d9aff3cce300 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 08:18:56 dist-test-slave-d3n8 krb5kdc[5772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:18:56.095742 5863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50cfc77216e3413a8673d9aff3cce300 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cfc77216e3413a8673d9aff3cce300" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 45811 } }
I20260619 08:18:56.095829 5863 leader_election.cc:302] T 00000000000000000000000000000000 P 50cfc77216e3413a8673d9aff3cce300 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50cfc77216e3413a8673d9aff3cce300; no voters:
I20260619 08:18:56.096063 5863 leader_election.cc:288] T 00000000000000000000000000000000 P 50cfc77216e3413a8673d9aff3cce300 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:18:56.096103 5868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50cfc77216e3413a8673d9aff3cce300 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:18:56.096272 5868 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50cfc77216e3413a8673d9aff3cce300 [term 1 LEADER]: Becoming Leader. State: Replica: 50cfc77216e3413a8673d9aff3cce300, State: Running, Role: LEADER
I20260619 08:18:56.096346 5868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50cfc77216e3413a8673d9aff3cce300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cfc77216e3413a8673d9aff3cce300" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 45811 } }
I20260619 08:18:56.096382 5863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50cfc77216e3413a8673d9aff3cce300 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:18:56.096660 5868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cfc77216e3413a8673d9aff3cce300 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50cfc77216e3413a8673d9aff3cce300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cfc77216e3413a8673d9aff3cce300" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 45811 } } }
I20260619 08:18:56.096727 5868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cfc77216e3413a8673d9aff3cce300 [sys.catalog]: This master's current role is: LEADER
I20260619 08:18:56.096839 5868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cfc77216e3413a8673d9aff3cce300 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50cfc77216e3413a8673d9aff3cce300. Latest consensus state: current_term: 1 leader_uuid: "50cfc77216e3413a8673d9aff3cce300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cfc77216e3413a8673d9aff3cce300" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 45811 } } }
I20260619 08:18:56.096895 5868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cfc77216e3413a8673d9aff3cce300 [sys.catalog]: This master's current role is: LEADER
I20260619 08:18:56.097019 5873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:18:56.097585 5873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:18:56.099377 5873 catalog_manager.cc:1347] Generated new cluster ID: e6ba19133b534e14a09594e6d2e8d5d5
I20260619 08:18:56.099426 5873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:18:56.100239 5866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.089258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36788 (local address 127.4.151.190:45811)
0619 08:18:56.089467 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:56.089475 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:56.089500 (+ 25us) server_negotiation.cc:407] Connection header received
0619 08:18:56.089905 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:56.089911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:56.089955 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:56.090156 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:56.090762 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.091781 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.092687 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.093091 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.093200 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.095746 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:56.095750 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:56.095750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:56.095762 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:56.098022 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:56.098446 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:56.098449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:56.098449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:56.098541 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:56.098766 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:56.098767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:56.098767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:56.099070 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:56.099206 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:56.099904 (+ 698us) server_negotiation.cc:299] Negotiation successful
0619 08:18:56.100045 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":90,"threads_started":1}
I20260619 08:18:56.108714 5873 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:18:56.109298 5873 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:18:56.122834 5873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50cfc77216e3413a8673d9aff3cce300: Generated new TSK 0
I20260619 08:18:56.123080 5873 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.151.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
I20260619 08:18:56.156613 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:45811
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:18:56.240736 5889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:18:56.241026 5889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:18:56.241056 5889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:18:56.242308 5889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:18:56.242350 5889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:18:56.242372 5889 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 08:18:56.242388 5889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:18:56.243822 5889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.151.190:45811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:18:56.244076 5889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:18:56.244298 5889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:18:56.249259 5895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:56.249285 5894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:56.249259 5897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:18:56.249647 5889 server_base.cc:1034] running on GCE node
I20260619 08:18:56.249832 5889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:18:56.250083 5889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:18:56.251250 5889 hybrid_clock.cc:648] HybridClock initialized: now 1781857136251237 us; error 29 us; skew 500 ppm
Jun 19 08:18:56 dist-test-slave-d3n8 krb5kdc[5772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857136, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:18:56.254029 5889 init.cc:377] Logged in from keytab as kudu/127.4.151.129@KRBTEST.COM (short username kudu)
I20260619 08:18:56.254572 5889 webserver.cc:460] Webserver started at http://127.4.151.129:44163/ using document root <none> and password file <none>
I20260619 08:18:56.254797 5889 fs_manager.cc:359] Metadata directory not provided
I20260619 08:18:56.254863 5889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:18:56.255012 5889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:18:56.256063 5889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "a3074c15b6234b4fa904343df8d5be42"
format_stamp: "Formatted at 2026-06-19 08:18:56 on dist-test-slave-d3n8"
server_key: "8e42d24d2487b70d14c5dc7148325208"
server_key_iv: "888029657e7f61e7d37c723c47a100f5"
server_key_version: "encryptionkey@0"
I20260619 08:18:56.256454 5889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "a3074c15b6234b4fa904343df8d5be42"
format_stamp: "Formatted at 2026-06-19 08:18:56 on dist-test-slave-d3n8"
server_key: "8e42d24d2487b70d14c5dc7148325208"
server_key_iv: "888029657e7f61e7d37c723c47a100f5"
server_key_version: "encryptionkey@0"
I20260619 08:18:56.258067 5889 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:18:56.259030 5904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:56.259267 5889 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:18:56.259383 5889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "a3074c15b6234b4fa904343df8d5be42"
format_stamp: "Formatted at 2026-06-19 08:18:56 on dist-test-slave-d3n8"
server_key: "8e42d24d2487b70d14c5dc7148325208"
server_key_iv: "888029657e7f61e7d37c723c47a100f5"
server_key_version: "encryptionkey@0"
I20260619 08:18:56.259459 5889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:18:56.292193 5889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:18:56.294497 5889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:18:56.294656 5889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:18:56.295207 5889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:18:56.295265 5889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:56.295312 5889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:18:56.295362 5889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 08:18:56 dist-test-slave-d3n8 krb5kdc[5772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857136, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:18:56.303112 5889 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:41873
I20260619 08:18:56.303257 6019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:41873 every 8 connection(s)
I20260619 08:18:56.303506 5889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
I20260619 08:18:56.306345 5866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.295837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:34925 (local address 127.4.151.190:45811)
0619 08:18:56.295897 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:56.295899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:56.296291 (+ 392us) server_negotiation.cc:407] Connection header received
0619 08:18:56.297082 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:56.297084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:56.297100 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:56.297174 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:56.298229 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.299124 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.300255 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.300641 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.300720 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.303406 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:56.303408 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:56.303409 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:56.303425 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:56.304974 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:56.305371 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:56.305418 (+ 47us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:56.305419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:56.305491 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:56.305704 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:56.305705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:56.305705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:56.305840 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:56.305895 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:56.306206 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 08:18:56.306281 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 08:18:56.306438 5913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.295827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:45811 (local address 127.4.151.129:34925)
0619 08:18:56.296128 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:56.296170 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:56.296929 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:56.297331 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:56.297335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:56.297778 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:18:56.298112 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.298120 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.299252 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.299252 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.300148 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.300150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.300756 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.300757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.300928 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.301560 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:18:56.301571 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:18:56.303311 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:18:56.305051 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:56.305057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:56.305059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:56.305274 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:56.305551 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:56.305551 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:56.305552 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:56.305647 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:56.305895 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:18:56.305899 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:18:56.306037 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 08:18:56.306124 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 08:18:56.306246 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":100,"threads_started":1}
I20260619 08:18:56.307017 6020 heartbeater.cc:347] Connected to a master server at 127.4.151.190:45811
I20260619 08:18:56.307107 6020 heartbeater.cc:464] Registering TS with master...
I20260619 08:18:56.307305 6020 heartbeater.cc:511] Master 127.4.151.190:45811 requested a full tablet report, sending...
I20260619 08:18:56.307853 5817 ts_manager.cc:194] Registered new tserver with Master: a3074c15b6234b4fa904343df8d5be42 (127.4.151.129:41873)
I20260619 08:18:56.308816 5817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.129@KRBTEST.COM'} at 127.4.151.129:34925
I20260619 08:18:56.312098 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 5889
I20260619 08:18:56.312209 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:18:56.312378 4702 external_mini_cluster.cc:1428] Setting key a468f8670ead9d273eeff65b62187822
I20260619 08:18:56.314178 5866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.308195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:37321 (local address 127.4.151.190:45811)
0619 08:18:56.308236 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:56.308238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:56.308407 (+ 169us) server_negotiation.cc:407] Connection header received
0619 08:18:56.308427 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:56.308428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:56.308436 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:56.308475 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:56.308811 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.309539 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.310197 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.310374 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.310431 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.311430 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:56.311432 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:56.311432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:56.311438 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:56.312904 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:56.313441 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:56.313442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:56.313442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:56.313504 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:56.313698 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:56.313699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:56.313699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:56.313790 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:56.313852 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:56.314062 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 08:18:56.314120 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:18:56.314273 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.308123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:45811 (local address 127.4.151.129:37321)
0619 08:18:56.308300 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:56.308311 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:56.308373 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:56.308479 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:56.308480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:56.308673 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:18:56.308741 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.308742 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.309622 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.309622 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.310118 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.310119 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.310437 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.310438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.310497 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.310926 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:18:56.310931 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:18:56.311336 (+ 405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:18:56.313083 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:56.313084 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:56.313085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:56.313321 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:56.313565 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:56.313566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:56.313567 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:56.313638 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:56.313855 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:18:56.313856 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:18:56.313916 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 08:18:56.313993 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 08:18:56.314112 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.4.151.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
I20260619 08:18:56.341755 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:45811
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:18:56.424163 6029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:18:56.424384 6029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:18:56.424409 6029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:18:56.425642 6029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:18:56.425683 6029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:18:56.425704 6029 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 08:18:56.425722 6029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:18:56.427186 6029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.151.190:45811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:18:56.427438 6029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:18:56.427678 6029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:18:56.432700 6034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:56.432721 6035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:56.432708 6037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:18:56.433115 6029 server_base.cc:1034] running on GCE node
I20260619 08:18:56.433331 6029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:18:56.433596 6029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:18:56.434749 6029 hybrid_clock.cc:648] HybridClock initialized: now 1781857136434730 us; error 39 us; skew 500 ppm
Jun 19 08:18:56 dist-test-slave-d3n8 krb5kdc[5772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857136, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:18:56.437282 6029 init.cc:377] Logged in from keytab as kudu/127.4.151.130@KRBTEST.COM (short username kudu)
I20260619 08:18:56.437744 6029 webserver.cc:460] Webserver started at http://127.4.151.130:36657/ using document root <none> and password file <none>
I20260619 08:18:56.438010 6029 fs_manager.cc:359] Metadata directory not provided
I20260619 08:18:56.438081 6029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:18:56.438218 6029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:18:56.439288 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "fcfbd0af911b4ba4ae77012eb6fb3286"
format_stamp: "Formatted at 2026-06-19 08:18:56 on dist-test-slave-d3n8"
server_key: "af3d39e626668775b7c3dcaab12a9178"
server_key_iv: "a966d1da97eb168cfb281c17f6a477bb"
server_key_version: "encryptionkey@0"
I20260619 08:18:56.439651 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "fcfbd0af911b4ba4ae77012eb6fb3286"
format_stamp: "Formatted at 2026-06-19 08:18:56 on dist-test-slave-d3n8"
server_key: "af3d39e626668775b7c3dcaab12a9178"
server_key_iv: "a966d1da97eb168cfb281c17f6a477bb"
server_key_version: "encryptionkey@0"
I20260619 08:18:56.441058 6029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:18:56.441992 6044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:56.442220 6029 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 08:18:56.442322 6029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "fcfbd0af911b4ba4ae77012eb6fb3286"
format_stamp: "Formatted at 2026-06-19 08:18:56 on dist-test-slave-d3n8"
server_key: "af3d39e626668775b7c3dcaab12a9178"
server_key_iv: "a966d1da97eb168cfb281c17f6a477bb"
server_key_version: "encryptionkey@0"
I20260619 08:18:56.442391 6029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:18:56.461736 6029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:18:56.464026 6029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:18:56.464202 6029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:18:56.464848 6029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:18:56.464900 6029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:56.464944 6029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:18:56.465008 6029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 08:18:56 dist-test-slave-d3n8 krb5kdc[5772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857136, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:18:56.472267 6029 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:35729
I20260619 08:18:56.472363 6159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:35729 every 8 connection(s)
I20260619 08:18:56.472675 6029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
I20260619 08:18:56.475406 5866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.465325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:51989 (local address 127.4.151.190:45811)
0619 08:18:56.465397 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:56.465400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:56.465754 (+ 354us) server_negotiation.cc:407] Connection header received
0619 08:18:56.466485 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:56.466487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:56.466504 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:56.466581 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:56.467570 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.468131 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.469788 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.470017 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.470095 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.472555 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:56.472556 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:56.472556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:56.472565 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:56.474037 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:56.474447 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:56.474449 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:56.474449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:56.474505 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:56.474773 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:56.474774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:56.474774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:56.474894 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:56.474970 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:56.475235 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 08:18:56.475315 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 08:18:56.475597 6053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.465331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:45811 (local address 127.4.151.130:51989)
0619 08:18:56.465541 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:56.465592 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:56.466326 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:56.466687 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:56.466693 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:56.467133 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:18:56.467446 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.467454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.468249 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.468251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.469677 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.469679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.470132 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.470134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.470263 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.470810 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:18:56.470819 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:18:56.472468 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:18:56.474134 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:56.474140 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:56.474142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:56.474357 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:56.474600 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:56.474601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:56.474601 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:56.474708 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:56.474960 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:18:56.474966 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:18:56.475110 (+ 144us) client_negotiation.cc:769] Sending connection context
0619 08:18:56.475215 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 08:18:56.475368 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":120,"threads_started":1}
I20260619 08:18:56.476178 6160 heartbeater.cc:347] Connected to a master server at 127.4.151.190:45811
I20260619 08:18:56.476289 6160 heartbeater.cc:464] Registering TS with master...
I20260619 08:18:56.476502 6160 heartbeater.cc:511] Master 127.4.151.190:45811 requested a full tablet report, sending...
I20260619 08:18:56.476866 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 6029
I20260619 08:18:56.476919 5816 ts_manager.cc:194] Registered new tserver with Master: fcfbd0af911b4ba4ae77012eb6fb3286 (127.4.151.130:35729)
I20260619 08:18:56.476969 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:18:56.477166 4702 external_mini_cluster.cc:1428] Setting key 851713cc0c4cad5f9de9f6809b00bb52
I20260619 08:18:56.477303 5816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.130@KRBTEST.COM'} at 127.4.151.130:51989
I20260619 08:18:56.484522 6166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.477524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:45811 (local address 127.4.151.130:44795)
0619 08:18:56.478147 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:56.478157 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:56.478207 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:56.478335 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:56.478335 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:56.478498 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:18:56.478581 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.478583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.479630 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.479631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.480227 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.480229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.480658 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.480659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.480734 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.481200 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:18:56.481205 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:18:56.481603 (+ 398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:18:56.483398 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:56.483400 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:56.483400 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:56.483576 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:56.483796 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:56.483797 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:56.483797 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:56.483873 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:56.484163 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:18:56.484164 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:18:56.484211 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 08:18:56.484306 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 08:18:56.484412 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":96,"threads_started":1}
I20260619 08:18:56.484551 5866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.477571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:44795 (local address 127.4.151.190:45811)
0619 08:18:56.477612 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:56.477615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:56.478203 (+ 588us) server_negotiation.cc:407] Connection header received
0619 08:18:56.478248 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:56.478249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:56.478260 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:56.478333 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:56.478654 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.479515 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.480389 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.480585 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.480661 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.481707 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:56.481709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:56.481709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:56.481716 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:56.483279 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:56.483663 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:56.483664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:56.483665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:56.483720 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:56.483940 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:56.483941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:56.483941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:56.484090 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:56.484162 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:56.484383 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 08:18:56.484463 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for kudu/127.4.151.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
I20260619 08:18:56.505548 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:45811
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:18:56.587476 6169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:18:56.587713 6169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:18:56.587756 6169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:18:56.588950 6169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:18:56.589012 6169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:18:56.589046 6169 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 08:18:56.589085 6169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:18:56.590559 6169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.151.190:45811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:18:56.590822 6169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:18:56.591051 6169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:18:56.596246 6177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:56.596276 6175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:56.596246 6174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:18:56.596450 6169 server_base.cc:1034] running on GCE node
I20260619 08:18:56.596637 6169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:18:56.596912 6169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:18:56.598098 6169 hybrid_clock.cc:648] HybridClock initialized: now 1781857136598083 us; error 48 us; skew 500 ppm
Jun 19 08:18:56 dist-test-slave-d3n8 krb5kdc[5772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857136, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:18:56.600572 6169 init.cc:377] Logged in from keytab as kudu/127.4.151.131@KRBTEST.COM (short username kudu)
I20260619 08:18:56.601006 6169 webserver.cc:460] Webserver started at http://127.4.151.131:45267/ using document root <none> and password file <none>
I20260619 08:18:56.601203 6169 fs_manager.cc:359] Metadata directory not provided
I20260619 08:18:56.601244 6169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:18:56.601332 6169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:18:56.602391 6169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "2e1841f31d43409792c0c0eb2fc8d74e"
format_stamp: "Formatted at 2026-06-19 08:18:56 on dist-test-slave-d3n8"
server_key: "4173c1756bf54c0890b7dbc0ce41364d"
server_key_iv: "f3d3ff8896dacf5f699f9457278ab04e"
server_key_version: "encryptionkey@0"
I20260619 08:18:56.602705 6169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "2e1841f31d43409792c0c0eb2fc8d74e"
format_stamp: "Formatted at 2026-06-19 08:18:56 on dist-test-slave-d3n8"
server_key: "4173c1756bf54c0890b7dbc0ce41364d"
server_key_iv: "f3d3ff8896dacf5f699f9457278ab04e"
server_key_version: "encryptionkey@0"
I20260619 08:18:56.604060 6169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:18:56.604785 6184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:56.605034 6169 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:18:56.605139 6169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "2e1841f31d43409792c0c0eb2fc8d74e"
format_stamp: "Formatted at 2026-06-19 08:18:56 on dist-test-slave-d3n8"
server_key: "4173c1756bf54c0890b7dbc0ce41364d"
server_key_iv: "f3d3ff8896dacf5f699f9457278ab04e"
server_key_version: "encryptionkey@0"
I20260619 08:18:56.605216 6169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:18:56.620986 6169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:18:56.623458 6169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:18:56.623631 6169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:18:56.624261 6169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:18:56.624312 6169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:56.624353 6169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:18:56.624367 6169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 08:18:56 dist-test-slave-d3n8 krb5kdc[5772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857136, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:18:56.632781 6169 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:36799
I20260619 08:18:56.632881 6299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:36799 every 8 connection(s)
I20260619 08:18:56.633219 6169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
I20260619 08:18:56.635263 5866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.624673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:47435 (local address 127.4.151.190:45811)
0619 08:18:56.624741 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:56.624743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:56.625026 (+ 283us) server_negotiation.cc:407] Connection header received
0619 08:18:56.625779 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:56.625781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:56.625797 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:56.625871 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:56.626794 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.627333 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.629146 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.629364 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.629478 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.632407 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:56.632409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:56.632409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:56.632416 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:56.634039 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:56.634416 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:56.634417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:56.634417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:56.634471 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:56.634658 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:56.634658 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:56.634659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:56.634770 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:56.634820 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:56.635118 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 08:18:56.635194 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 08:18:56.635340 6193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.624667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:45811 (local address 127.4.151.131:47435)
0619 08:18:56.624834 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:56.624889 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:56.625648 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:56.625965 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:56.625970 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:56.626354 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:18:56.626688 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.626695 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.627507 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.627509 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.629052 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.629056 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.629675 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.629677 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.629852 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.630491 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:18:56.630502 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:18:56.632299 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:18:56.634116 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:56.634121 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:56.634124 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:56.634333 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:56.634525 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:56.634525 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:56.634526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:56.634611 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:56.634823 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:18:56.634827 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:18:56.634970 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 08:18:56.635059 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 08:18:56.635179 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260619 08:18:56.635923 6300 heartbeater.cc:347] Connected to a master server at 127.4.151.190:45811
I20260619 08:18:56.636106 6300 heartbeater.cc:464] Registering TS with master...
I20260619 08:18:56.636322 6300 heartbeater.cc:511] Master 127.4.151.190:45811 requested a full tablet report, sending...
I20260619 08:18:56.636754 5817 ts_manager.cc:194] Registered new tserver with Master: 2e1841f31d43409792c0c0eb2fc8d74e (127.4.151.131:36799)
I20260619 08:18:56.637287 5817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.131@KRBTEST.COM'} at 127.4.151.131:47435
I20260619 08:18:56.640803 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 6169
I20260619 08:18:56.640903 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:18:56.641078 4702 external_mini_cluster.cc:1428] Setting key 6b59eb5f41df6622ba9df1eae46b1c67
I20260619 08:18:56.642189 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:18:56.643069 5866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.637015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:39977 (local address 127.4.151.190:45811)
0619 08:18:56.637058 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:56.637060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:56.637110 (+ 50us) server_negotiation.cc:407] Connection header received
0619 08:18:56.637168 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:56.637169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:56.637177 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:56.637224 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:56.637627 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.638168 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.638841 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.639041 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.639100 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.640071 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:56.640072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:56.640072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:56.640079 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:56.641878 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:56.642295 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:56.642296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:56.642296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:56.642358 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:56.642544 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:56.642545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:56.642546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:56.642662 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:56.642733 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:56.642889 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 08:18:56.642965 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:18:56.643079 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.636910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:45811 (local address 127.4.151.131:39977)
0619 08:18:56.637046 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:56.637058 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:56.637115 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:56.637229 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:56.637229 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:56.637465 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:18:56.637553 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.637555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.638253 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.638254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.638748 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.638750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.639161 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.639162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.639228 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.639660 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:18:56.639664 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:18:56.639943 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:18:56.641971 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:56.641972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:56.641973 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:56.642216 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:56.642417 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:56.642418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:56.642419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:56.642478 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:56.642724 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:18:56.642725 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:18:56.642773 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 08:18:56.642859 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 08:18:56.642967 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
Jun 19 08:18:56 dist-test-slave-d3n8 krb5kdc[5772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857136, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 08:18:56 dist-test-slave-d3n8 krb5kdc[5772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857136, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:18:56.667109 5866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.659534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36804 (local address 127.4.151.190:45811)
0619 08:18:56.659603 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:56.659605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:56.659612 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:18:56.659635 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:56.659636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:56.659645 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:56.659713 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:56.660362 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.660872 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.661669 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.661857 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.661935 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.664524 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:56.664525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:56.664525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:56.664533 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:56.666024 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:56.666333 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:56.666334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:56.666334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:56.666383 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:56.666651 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:56.666652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:56.666652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:56.666763 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:56.666841 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:56.666961 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 08:18:56.667020 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 08:18:56.669044 5817 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:18:56.670818 5817 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:18:56.677073 6317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.673133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:41873 (local address 127.0.0.1:36024)
0619 08:18:56.673304 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:56.673346 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:56.673435 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:56.673689 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:56.673692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:56.673694 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:18:56.673768 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.673770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.674811 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.674812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.675841 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.675844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.676788 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.676789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.676870 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.676886 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 08:18:56.676934 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 08:18:56.676985 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":26,"thread_start_us":76,"threads_started":1}
I20260619 08:18:56.677511 6320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.673322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36024 (local address 127.4.151.129:41873)
0619 08:18:56.673516 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:56.673524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:56.673535 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:18:56.673576 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:56.673579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:56.673596 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:56.673684 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:18:56.673851 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.674726 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.675954 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.676689 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.677213 (+ 524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.677283 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:56.677310 (+ 27us) server_negotiation.cc:299] Negotiation successful
0619 08:18:56.677351 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":90,"threads_started":1}
I20260619 08:18:56.678406 6318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.673331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:36799 (local address 127.0.0.1:54700)
0619 08:18:56.673555 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:56.673564 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:56.673603 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:56.675223 (+ 1620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:56.675223 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:56.675225 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:18:56.675286 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.675287 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.676508 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.676509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.677348 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.677350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.678140 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.678141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.678216 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.678221 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:18:56.678281 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 08:18:56.678328 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":2,"thread_start_us":116,"threads_started":1}
I20260619 08:18:56.678297 5954 tablet_service.cc:1467] Processing CreateTablet for tablet 9e2a750b3d19445ab032cac35c315e50 (DEFAULT_TABLE table=test-table [id=55c5e3e4606544ba95d8959868dd48a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:18:56.678581 6319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.673234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43586 (local address 127.4.151.130:35729)
0619 08:18:56.673737 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:56.673741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:56.673750 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:18:56.673791 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:56.673792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:56.673803 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:56.673985 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:18:56.674048 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.674790 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.676386 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.677162 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.677226 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.677284 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:56.678396 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0619 08:18:56.678446 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":99,"threads_started":1}
I20260619 08:18:56.678690 5954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e2a750b3d19445ab032cac35c315e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:18:56.678715 6322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.673582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54700 (local address 127.4.151.131:36799)
0619 08:18:56.675056 (+ 1474us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:56.675061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:56.675071 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:18:56.675108 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:56.675110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:56.675126 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:56.675208 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:18:56.675374 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.676427 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.677468 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.678057 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.678117 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.678150 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:56.678571 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 08:18:56.678615 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1432,"thread_start_us":82,"threads_started":1}
I20260619 08:18:56.678992 6321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.673133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:35729 (local address 127.0.0.1:43586)
0619 08:18:56.673497 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:56.673508 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:56.673569 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:56.673901 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:56.673902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:56.673904 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:18:56.673971 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.673972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.674886 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.674887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.676301 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.676303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.678182 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.678182 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.678288 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.678292 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:18:56.678886 (+ 594us) client_negotiation.cc:240] Negotiation successful
0619 08:18:56.678933 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":3,"thread_start_us":64,"threads_started":1}
I20260619 08:18:56.679261 6234 tablet_service.cc:1467] Processing CreateTablet for tablet 9e2a750b3d19445ab032cac35c315e50 (DEFAULT_TABLE table=test-table [id=55c5e3e4606544ba95d8959868dd48a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:18:56.679595 6234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e2a750b3d19445ab032cac35c315e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:18:56.679632 6094 tablet_service.cc:1467] Processing CreateTablet for tablet 9e2a750b3d19445ab032cac35c315e50 (DEFAULT_TABLE table=test-table [id=55c5e3e4606544ba95d8959868dd48a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:18:56.679975 6094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e2a750b3d19445ab032cac35c315e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:18:56.682708 6323 tablet_bootstrap.cc:492] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42: Bootstrap starting.
I20260619 08:18:56.682988 6325 tablet_bootstrap.cc:492] T 9e2a750b3d19445ab032cac35c315e50 P 2e1841f31d43409792c0c0eb2fc8d74e: Bootstrap starting.
I20260619 08:18:56.683411 6323 tablet_bootstrap.cc:654] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42: Neither blocks nor log segments found. Creating new log.
I20260619 08:18:56.683696 6323 log.cc:824] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42: Log is configured to *not* fsync() on all Append() calls
I20260619 08:18:56.683804 6325 tablet_bootstrap.cc:654] T 9e2a750b3d19445ab032cac35c315e50 P 2e1841f31d43409792c0c0eb2fc8d74e: Neither blocks nor log segments found. Creating new log.
I20260619 08:18:56.684108 6325 log.cc:824] T 9e2a750b3d19445ab032cac35c315e50 P 2e1841f31d43409792c0c0eb2fc8d74e: Log is configured to *not* fsync() on all Append() calls
I20260619 08:18:56.684388 6323 tablet_bootstrap.cc:492] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42: No bootstrap required, opened a new log
I20260619 08:18:56.684435 6324 tablet_bootstrap.cc:492] T 9e2a750b3d19445ab032cac35c315e50 P fcfbd0af911b4ba4ae77012eb6fb3286: Bootstrap starting.
I20260619 08:18:56.684466 6323 ts_tablet_manager.cc:1397] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:18:56.684931 6324 tablet_bootstrap.cc:654] T 9e2a750b3d19445ab032cac35c315e50 P fcfbd0af911b4ba4ae77012eb6fb3286: Neither blocks nor log segments found. Creating new log.
I20260619 08:18:56.685281 6324 log.cc:824] T 9e2a750b3d19445ab032cac35c315e50 P fcfbd0af911b4ba4ae77012eb6fb3286: Log is configured to *not* fsync() on all Append() calls
I20260619 08:18:56.685751 6325 tablet_bootstrap.cc:492] T 9e2a750b3d19445ab032cac35c315e50 P 2e1841f31d43409792c0c0eb2fc8d74e: No bootstrap required, opened a new log
I20260619 08:18:56.685868 6325 ts_tablet_manager.cc:1397] T 9e2a750b3d19445ab032cac35c315e50 P 2e1841f31d43409792c0c0eb2fc8d74e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 08:18:56.685947 6324 tablet_bootstrap.cc:492] T 9e2a750b3d19445ab032cac35c315e50 P fcfbd0af911b4ba4ae77012eb6fb3286: No bootstrap required, opened a new log
I20260619 08:18:56.686000 6324 ts_tablet_manager.cc:1397] T 9e2a750b3d19445ab032cac35c315e50 P fcfbd0af911b4ba4ae77012eb6fb3286: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:18:56.686642 6323 raft_consensus.cc:348] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:56.686816 6323 raft_consensus.cc:374] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:18:56.686852 6323 raft_consensus.cc:729] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3074c15b6234b4fa904343df8d5be42, State: Initialized, Role: FOLLOWER
I20260619 08:18:56.687011 6323 consensus_queue.cc:260] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:56.687288 6323 ts_tablet_manager.cc:1428] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:18:56.687399 6020 heartbeater.cc:503] Master 127.4.151.190:45811 was elected leader, sending a full tablet report...
I20260619 08:18:56.687592 6324 raft_consensus.cc:348] T 9e2a750b3d19445ab032cac35c315e50 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:56.687739 6324 raft_consensus.cc:374] T 9e2a750b3d19445ab032cac35c315e50 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:18:56.687779 6324 raft_consensus.cc:729] T 9e2a750b3d19445ab032cac35c315e50 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcfbd0af911b4ba4ae77012eb6fb3286, State: Initialized, Role: FOLLOWER
I20260619 08:18:56.687891 6324 consensus_queue.cc:260] T 9e2a750b3d19445ab032cac35c315e50 P fcfbd0af911b4ba4ae77012eb6fb3286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:56.688031 6325 raft_consensus.cc:348] T 9e2a750b3d19445ab032cac35c315e50 P 2e1841f31d43409792c0c0eb2fc8d74e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:56.688158 6324 ts_tablet_manager.cc:1428] T 9e2a750b3d19445ab032cac35c315e50 P fcfbd0af911b4ba4ae77012eb6fb3286: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:18:56.688179 6160 heartbeater.cc:503] Master 127.4.151.190:45811 was elected leader, sending a full tablet report...
I20260619 08:18:56.688185 6325 raft_consensus.cc:374] T 9e2a750b3d19445ab032cac35c315e50 P 2e1841f31d43409792c0c0eb2fc8d74e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:18:56.688220 6325 raft_consensus.cc:729] T 9e2a750b3d19445ab032cac35c315e50 P 2e1841f31d43409792c0c0eb2fc8d74e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e1841f31d43409792c0c0eb2fc8d74e, State: Initialized, Role: FOLLOWER
I20260619 08:18:56.688323 6325 consensus_queue.cc:260] T 9e2a750b3d19445ab032cac35c315e50 P 2e1841f31d43409792c0c0eb2fc8d74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:56.688566 6325 ts_tablet_manager.cc:1428] T 9e2a750b3d19445ab032cac35c315e50 P 2e1841f31d43409792c0c0eb2fc8d74e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 08:18:56.688687 6300 heartbeater.cc:503] Master 127.4.151.190:45811 was elected leader, sending a full tablet report...
I20260619 08:18:56.689206 6329 raft_consensus.cc:482] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:18:56.689293 6329 raft_consensus.cc:504] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:56.689637 6329 leader_election.cc:288] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcfbd0af911b4ba4ae77012eb6fb3286 (127.4.151.130:35729), 2e1841f31d43409792c0c0eb2fc8d74e (127.4.151.131:36799)
I20260619 08:18:56.692512 5913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.689775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:35729 (local address 127.4.151.129:49795)
0619 08:18:56.689816 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:56.689827 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:56.689873 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:56.690246 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:56.690247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:56.690249 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:18:56.690305 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.690306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.690930 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.690931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.691691 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.691693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.692239 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.692240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.692327 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.692331 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:18:56.692391 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 08:18:56.692436 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":19}
I20260619 08:18:56.692502 6319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.690079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:49795 (local address 127.4.151.130:35729)
0619 08:18:56.690125 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:56.690126 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:56.690134 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:18:56.690153 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:56.690153 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:56.690161 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:56.690230 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:18:56.690373 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.690841 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.691779 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.692155 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.692232 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.692241 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:56.692395 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 08:18:56.692429 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 08:18:56.692507 6322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.689973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:43241 (local address 127.4.151.131:36799)
0619 08:18:56.690011 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:56.690013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:56.690073 (+ 60us) server_negotiation.cc:407] Connection header received
0619 08:18:56.690135 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:56.690136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:56.690145 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:56.690200 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:18:56.690357 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.690823 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.691778 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.692153 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.692235 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.692242 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:56.692407 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 08:18:56.692440 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:18:56.692552 6332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.689892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:36799 (local address 127.4.151.129:43241)
0619 08:18:56.690016 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:56.690028 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:56.690086 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:56.690203 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:56.690204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:56.690205 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:18:56.690280 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.690282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.690916 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.690917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.691693 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.691695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.692240 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.692241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.692336 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.692339 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:18:56.692401 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 08:18:56.692440 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":28,"thread_start_us":84,"threads_started":1}
I20260619 08:18:56.693166 6114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e2a750b3d19445ab032cac35c315e50" candidate_uuid: "a3074c15b6234b4fa904343df8d5be42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" is_pre_election: true
I20260619 08:18:56.693166 6254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e2a750b3d19445ab032cac35c315e50" candidate_uuid: "a3074c15b6234b4fa904343df8d5be42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" is_pre_election: true
I20260619 08:18:56.693295 6114 raft_consensus.cc:2454] T 9e2a750b3d19445ab032cac35c315e50 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3074c15b6234b4fa904343df8d5be42 in term 0.
I20260619 08:18:56.693295 6254 raft_consensus.cc:2454] T 9e2a750b3d19445ab032cac35c315e50 P 2e1841f31d43409792c0c0eb2fc8d74e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3074c15b6234b4fa904343df8d5be42 in term 0.
I20260619 08:18:56.693532 5907 leader_election.cc:302] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3074c15b6234b4fa904343df8d5be42, fcfbd0af911b4ba4ae77012eb6fb3286; no voters:
I20260619 08:18:56.693650 6329 raft_consensus.cc:2781] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:18:56.693745 6329 raft_consensus.cc:482] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:18:56.693787 6329 raft_consensus.cc:3037] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:18:56.694531 6329 raft_consensus.cc:504] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:56.694679 6329 leader_election.cc:288] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42 [CANDIDATE]: Term 1 election: Requested vote from peers fcfbd0af911b4ba4ae77012eb6fb3286 (127.4.151.130:35729), 2e1841f31d43409792c0c0eb2fc8d74e (127.4.151.131:36799)
I20260619 08:18:56.694857 6114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e2a750b3d19445ab032cac35c315e50" candidate_uuid: "a3074c15b6234b4fa904343df8d5be42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286"
I20260619 08:18:56.694896 6254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e2a750b3d19445ab032cac35c315e50" candidate_uuid: "a3074c15b6234b4fa904343df8d5be42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e1841f31d43409792c0c0eb2fc8d74e"
I20260619 08:18:56.694980 6254 raft_consensus.cc:3037] T 9e2a750b3d19445ab032cac35c315e50 P 2e1841f31d43409792c0c0eb2fc8d74e [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:18:56.694934 6114 raft_consensus.cc:3037] T 9e2a750b3d19445ab032cac35c315e50 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:18:56.695719 6114 raft_consensus.cc:2454] T 9e2a750b3d19445ab032cac35c315e50 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3074c15b6234b4fa904343df8d5be42 in term 1.
I20260619 08:18:56.695719 6254 raft_consensus.cc:2454] T 9e2a750b3d19445ab032cac35c315e50 P 2e1841f31d43409792c0c0eb2fc8d74e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3074c15b6234b4fa904343df8d5be42 in term 1.
I20260619 08:18:56.695875 5907 leader_election.cc:302] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3074c15b6234b4fa904343df8d5be42, fcfbd0af911b4ba4ae77012eb6fb3286; no voters:
I20260619 08:18:56.695978 6329 raft_consensus.cc:2781] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:18:56.696173 6329 raft_consensus.cc:686] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42 [term 1 LEADER]: Becoming Leader. State: Replica: a3074c15b6234b4fa904343df8d5be42, State: Running, Role: LEADER
I20260619 08:18:56.696277 6329 consensus_queue.cc:237] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:56.697105 5816 catalog_manager.cc:5495] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3074c15b6234b4fa904343df8d5be42 (127.4.151.129). New cstate: current_term: 1 leader_uuid: "a3074c15b6234b4fa904343df8d5be42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } health_report { overall_health: UNKNOWN } } }
I20260619 08:18:56.706506 6320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.703585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36036 (local address 127.4.151.129:41873)
0619 08:18:56.703629 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:56.703631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:56.703641 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:18:56.703705 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:56.703706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:56.703723 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:56.703781 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:18:56.704058 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.704618 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.705643 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.705852 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.705930 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.706168 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:18:56.706264 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:18:56.706332 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:56.706410 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 08:18:56.706430 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:18:56.708865 6254 raft_consensus.cc:1261] T 9e2a750b3d19445ab032cac35c315e50 P 2e1841f31d43409792c0c0eb2fc8d74e [term 1 FOLLOWER]: Refusing update from remote peer a3074c15b6234b4fa904343df8d5be42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:18:56.709009 6114 raft_consensus.cc:1261] T 9e2a750b3d19445ab032cac35c315e50 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 1 FOLLOWER]: Refusing update from remote peer a3074c15b6234b4fa904343df8d5be42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:18:56.709268 6329 consensus_queue.cc:1035] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:18:56.709435 6333 consensus_queue.cc:1035] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:18:56.712476 6336 mvcc.cc:205] Tried to move back new op lower bound from 7298486831956615168 to 7298486831908577280. Current Snapshot: MvccSnapshot[applied={T|T < 7298486831956615168}]
I20260619 08:18:56.712512 6338 mvcc.cc:205] Tried to move back new op lower bound from 7298486831956615168 to 7298486831908577280. Current Snapshot: MvccSnapshot[applied={T|T < 7298486831956615168}]
I20260619 08:18:56.715971 5817 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:18:56.716084 5817 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:18:56.717000 5817 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50cfc77216e3413a8673d9aff3cce300: Sending DeleteTablet for 3 replicas of tablet 9e2a750b3d19445ab032cac35c315e50
I20260619 08:18:56.717404 6094 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e2a750b3d19445ab032cac35c315e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:18:56 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:43586
I20260619 08:18:56.717409 5954 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e2a750b3d19445ab032cac35c315e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:18:56 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:36024
I20260619 08:18:56.717451 6234 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e2a750b3d19445ab032cac35c315e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:18:56 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:54700
I20260619 08:18:56.717705 6345 tablet_replica.cc:330] T 9e2a750b3d19445ab032cac35c315e50 P fcfbd0af911b4ba4ae77012eb6fb3286: stopping tablet replica
I20260619 08:18:56.717819 6345 raft_consensus.cc:2229] T 9e2a750b3d19445ab032cac35c315e50 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:18:56.717927 6345 raft_consensus.cc:2258] T 9e2a750b3d19445ab032cac35c315e50 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:18:56.718142 5817 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:18:56.718290 5817 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:18:56.718883 6345 ts_tablet_manager.cc:1905] T 9e2a750b3d19445ab032cac35c315e50 P fcfbd0af911b4ba4ae77012eb6fb3286: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:18:56.719589 6094 tablet_service.cc:1467] Processing CreateTablet for tablet f06e423a36e1409b93fbdcd54bbade31 (DEFAULT_TABLE table=test-table [id=c813862968d54a68a2e5dd68cbf77a09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:18:56.719628 6234 tablet_service.cc:1467] Processing CreateTablet for tablet f06e423a36e1409b93fbdcd54bbade31 (DEFAULT_TABLE table=test-table [id=c813862968d54a68a2e5dd68cbf77a09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:18:56.719707 6094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f06e423a36e1409b93fbdcd54bbade31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:18:56.719729 6234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f06e423a36e1409b93fbdcd54bbade31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:18:56.721024 6345 ts_tablet_manager.cc:1918] T 9e2a750b3d19445ab032cac35c315e50 P fcfbd0af911b4ba4ae77012eb6fb3286: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:18:56.721093 6345 log.cc:1196] T 9e2a750b3d19445ab032cac35c315e50 P fcfbd0af911b4ba4ae77012eb6fb3286: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/9e2a750b3d19445ab032cac35c315e50
I20260619 08:18:56.721343 6345 ts_tablet_manager.cc:1939] T 9e2a750b3d19445ab032cac35c315e50 P fcfbd0af911b4ba4ae77012eb6fb3286: Deleting consensus metadata
I20260619 08:18:56.721489 6325 tablet_bootstrap.cc:492] T f06e423a36e1409b93fbdcd54bbade31 P 2e1841f31d43409792c0c0eb2fc8d74e: Bootstrap starting.
I20260619 08:18:56.721501 6324 tablet_bootstrap.cc:492] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286: Bootstrap starting.
I20260619 08:18:56.721885 6324 tablet_bootstrap.cc:654] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286: Neither blocks nor log segments found. Creating new log.
I20260619 08:18:56.721933 6325 tablet_bootstrap.cc:654] T f06e423a36e1409b93fbdcd54bbade31 P 2e1841f31d43409792c0c0eb2fc8d74e: Neither blocks nor log segments found. Creating new log.
I20260619 08:18:56.721954 5803 catalog_manager.cc:4841] TS fcfbd0af911b4ba4ae77012eb6fb3286 (127.4.151.130:35729): tablet 9e2a750b3d19445ab032cac35c315e50 (table test-table [id=55c5e3e4606544ba95d8959868dd48a5]) successfully deleted
I20260619 08:18:56.722338 5954 tablet_service.cc:1467] Processing CreateTablet for tablet f06e423a36e1409b93fbdcd54bbade31 (DEFAULT_TABLE table=test-table [id=c813862968d54a68a2e5dd68cbf77a09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:18:56.722476 5954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f06e423a36e1409b93fbdcd54bbade31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:18:56.722571 6324 tablet_bootstrap.cc:492] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286: No bootstrap required, opened a new log
I20260619 08:18:56.722612 6324 ts_tablet_manager.cc:1397] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:18:56.722687 6325 tablet_bootstrap.cc:492] T f06e423a36e1409b93fbdcd54bbade31 P 2e1841f31d43409792c0c0eb2fc8d74e: No bootstrap required, opened a new log
I20260619 08:18:56.722729 6325 ts_tablet_manager.cc:1397] T f06e423a36e1409b93fbdcd54bbade31 P 2e1841f31d43409792c0c0eb2fc8d74e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:18:56.722771 6324 raft_consensus.cc:348] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:56.722848 6324 raft_consensus.cc:374] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:18:56.722867 6324 raft_consensus.cc:729] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcfbd0af911b4ba4ae77012eb6fb3286, State: Initialized, Role: FOLLOWER
I20260619 08:18:56.722915 6324 consensus_queue.cc:260] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:56.722868 6325 raft_consensus.cc:348] T f06e423a36e1409b93fbdcd54bbade31 P 2e1841f31d43409792c0c0eb2fc8d74e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:56.722961 6325 raft_consensus.cc:374] T f06e423a36e1409b93fbdcd54bbade31 P 2e1841f31d43409792c0c0eb2fc8d74e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:18:56.722997 6325 raft_consensus.cc:729] T f06e423a36e1409b93fbdcd54bbade31 P 2e1841f31d43409792c0c0eb2fc8d74e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e1841f31d43409792c0c0eb2fc8d74e, State: Initialized, Role: FOLLOWER
I20260619 08:18:56.722998 6324 ts_tablet_manager.cc:1428] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:56.723050 6325 consensus_queue.cc:260] T f06e423a36e1409b93fbdcd54bbade31 P 2e1841f31d43409792c0c0eb2fc8d74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:56.723160 6325 ts_tablet_manager.cc:1428] T f06e423a36e1409b93fbdcd54bbade31 P 2e1841f31d43409792c0c0eb2fc8d74e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260619 08:18:56.723289 6161 tablet.cc:2369] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:18:56.723877 6323 tablet_bootstrap.cc:492] T f06e423a36e1409b93fbdcd54bbade31 P a3074c15b6234b4fa904343df8d5be42: Bootstrap starting.
I20260619 08:18:56.724287 6323 tablet_bootstrap.cc:654] T f06e423a36e1409b93fbdcd54bbade31 P a3074c15b6234b4fa904343df8d5be42: Neither blocks nor log segments found. Creating new log.
I20260619 08:18:56.724905 6323 tablet_bootstrap.cc:492] T f06e423a36e1409b93fbdcd54bbade31 P a3074c15b6234b4fa904343df8d5be42: No bootstrap required, opened a new log
I20260619 08:18:56.724969 6323 ts_tablet_manager.cc:1397] T f06e423a36e1409b93fbdcd54bbade31 P a3074c15b6234b4fa904343df8d5be42: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:18:56.725112 6323 raft_consensus.cc:348] T f06e423a36e1409b93fbdcd54bbade31 P a3074c15b6234b4fa904343df8d5be42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:56.725183 6323 raft_consensus.cc:374] T f06e423a36e1409b93fbdcd54bbade31 P a3074c15b6234b4fa904343df8d5be42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:18:56.725215 6323 raft_consensus.cc:729] T f06e423a36e1409b93fbdcd54bbade31 P a3074c15b6234b4fa904343df8d5be42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3074c15b6234b4fa904343df8d5be42, State: Initialized, Role: FOLLOWER
I20260619 08:18:56.725263 6323 consensus_queue.cc:260] T f06e423a36e1409b93fbdcd54bbade31 P a3074c15b6234b4fa904343df8d5be42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:56.725344 6323 ts_tablet_manager.cc:1428] T f06e423a36e1409b93fbdcd54bbade31 P a3074c15b6234b4fa904343df8d5be42: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 08:18:56.729521 6347 tablet_replica.cc:330] T 9e2a750b3d19445ab032cac35c315e50 P 2e1841f31d43409792c0c0eb2fc8d74e: stopping tablet replica
I20260619 08:18:56.729650 6347 raft_consensus.cc:2229] T 9e2a750b3d19445ab032cac35c315e50 P 2e1841f31d43409792c0c0eb2fc8d74e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:18:56.729780 6347 raft_consensus.cc:2258] T 9e2a750b3d19445ab032cac35c315e50 P 2e1841f31d43409792c0c0eb2fc8d74e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:18:56.730038 6346 tablet_replica.cc:330] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42: stopping tablet replica
I20260619 08:18:56.730147 6346 raft_consensus.cc:2229] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:18:56.730278 6346 raft_consensus.cc:2258] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:18:56.730707 6346 ts_tablet_manager.cc:1905] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:18:56.730707 6347 ts_tablet_manager.cc:1905] T 9e2a750b3d19445ab032cac35c315e50 P 2e1841f31d43409792c0c0eb2fc8d74e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:18:56.732458 6347 ts_tablet_manager.cc:1918] T 9e2a750b3d19445ab032cac35c315e50 P 2e1841f31d43409792c0c0eb2fc8d74e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:18:56.732458 6346 ts_tablet_manager.cc:1918] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:18:56.732537 6346 log.cc:1196] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-0/wal/wals/9e2a750b3d19445ab032cac35c315e50
I20260619 08:18:56.732537 6347 log.cc:1196] T 9e2a750b3d19445ab032cac35c315e50 P 2e1841f31d43409792c0c0eb2fc8d74e: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/9e2a750b3d19445ab032cac35c315e50
I20260619 08:18:56.732826 6347 ts_tablet_manager.cc:1939] T 9e2a750b3d19445ab032cac35c315e50 P 2e1841f31d43409792c0c0eb2fc8d74e: Deleting consensus metadata
I20260619 08:18:56.733127 6346 ts_tablet_manager.cc:1939] T 9e2a750b3d19445ab032cac35c315e50 P a3074c15b6234b4fa904343df8d5be42: Deleting consensus metadata
I20260619 08:18:56.733291 5803 catalog_manager.cc:4841] TS 2e1841f31d43409792c0c0eb2fc8d74e (127.4.151.131:36799): tablet 9e2a750b3d19445ab032cac35c315e50 (table test-table [id=55c5e3e4606544ba95d8959868dd48a5]) successfully deleted
I20260619 08:18:56.733603 5801 catalog_manager.cc:4841] TS a3074c15b6234b4fa904343df8d5be42 (127.4.151.129:41873): tablet 9e2a750b3d19445ab032cac35c315e50 (table test-table [id=55c5e3e4606544ba95d8959868dd48a5]) successfully deleted
W20260619 08:18:56.804239 6021 tablet.cc:2369] T f06e423a36e1409b93fbdcd54bbade31 P a3074c15b6234b4fa904343df8d5be42: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:18:56.883827 6301 tablet.cc:2369] T f06e423a36e1409b93fbdcd54bbade31 P 2e1841f31d43409792c0c0eb2fc8d74e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:18:56.921656 6330 raft_consensus.cc:482] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:18:56.921782 6330 raft_consensus.cc:504] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:56.922175 6330 leader_election.cc:288] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3074c15b6234b4fa904343df8d5be42 (127.4.151.129:41873), 2e1841f31d43409792c0c0eb2fc8d74e (127.4.151.131:36799)
I20260619 08:18:56.925238 6053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.922374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:41873 (local address 127.4.151.130:53647)
0619 08:18:56.922421 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:56.922437 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:56.922490 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:56.922648 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:56.922649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:56.922651 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:18:56.922733 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.922736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.923404 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.923405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.924390 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.924392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.924923 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.924924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.925025 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.925029 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:18:56.925091 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 08:18:56.925150 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 08:18:56.925266 6322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.922553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:33715 (local address 127.4.151.131:36799)
0619 08:18:56.922602 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:56.922604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:56.922614 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:18:56.922650 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:56.922651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:56.922660 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:56.922719 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:18:56.922910 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.923471 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.924494 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.924909 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.924985 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.924993 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:56.925155 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 08:18:56.925192 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 08:18:56.925244 6320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.922411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:53647 (local address 127.4.151.129:41873)
0619 08:18:56.922485 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:56.922487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:56.922499 (+ 12us) server_negotiation.cc:407] Connection header received
0619 08:18:56.922568 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:56.922569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:56.922582 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:56.922644 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:18:56.922819 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.923322 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.924496 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.924848 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.924904 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.924911 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:56.925117 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 08:18:56.925165 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:18:56.925577 6351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.922384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:36799 (local address 127.4.151.130:33715)
0619 08:18:56.922533 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:56.922545 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:56.922602 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:56.922740 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:56.922741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:56.922742 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:18:56.922836 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.922838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.923558 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.923559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.924395 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.924397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.925011 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.925012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.925061 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.925065 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:18:56.925145 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 08:18:56.925181 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":16,"thread_start_us":70,"threads_started":1}
I20260619 08:18:56.925681 5974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06e423a36e1409b93fbdcd54bbade31" candidate_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3074c15b6234b4fa904343df8d5be42" is_pre_election: true
I20260619 08:18:56.925824 5974 raft_consensus.cc:2454] T f06e423a36e1409b93fbdcd54bbade31 P a3074c15b6234b4fa904343df8d5be42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcfbd0af911b4ba4ae77012eb6fb3286 in term 0.
I20260619 08:18:56.925884 6254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06e423a36e1409b93fbdcd54bbade31" candidate_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" is_pre_election: true
I20260619 08:18:56.926004 6254 raft_consensus.cc:2454] T f06e423a36e1409b93fbdcd54bbade31 P 2e1841f31d43409792c0c0eb2fc8d74e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcfbd0af911b4ba4ae77012eb6fb3286 in term 0.
I20260619 08:18:56.926071 6045 leader_election.cc:302] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3074c15b6234b4fa904343df8d5be42, fcfbd0af911b4ba4ae77012eb6fb3286; no voters:
I20260619 08:18:56.926236 6330 raft_consensus.cc:2781] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:18:56.926277 6330 raft_consensus.cc:482] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:18:56.926292 6330 raft_consensus.cc:3037] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:18:56.926973 6330 raft_consensus.cc:504] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:56.927124 6330 leader_election.cc:288] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286 [CANDIDATE]: Term 1 election: Requested vote from peers a3074c15b6234b4fa904343df8d5be42 (127.4.151.129:41873), 2e1841f31d43409792c0c0eb2fc8d74e (127.4.151.131:36799)
I20260619 08:18:56.927289 6254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06e423a36e1409b93fbdcd54bbade31" candidate_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e1841f31d43409792c0c0eb2fc8d74e"
I20260619 08:18:56.927377 6254 raft_consensus.cc:3037] T f06e423a36e1409b93fbdcd54bbade31 P 2e1841f31d43409792c0c0eb2fc8d74e [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:18:56.927398 5974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06e423a36e1409b93fbdcd54bbade31" candidate_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3074c15b6234b4fa904343df8d5be42"
I20260619 08:18:56.927488 5974 raft_consensus.cc:3037] T f06e423a36e1409b93fbdcd54bbade31 P a3074c15b6234b4fa904343df8d5be42 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:18:56.928042 6254 raft_consensus.cc:2454] T f06e423a36e1409b93fbdcd54bbade31 P 2e1841f31d43409792c0c0eb2fc8d74e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcfbd0af911b4ba4ae77012eb6fb3286 in term 1.
I20260619 08:18:56.928215 6047 leader_election.cc:302] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e1841f31d43409792c0c0eb2fc8d74e, fcfbd0af911b4ba4ae77012eb6fb3286; no voters:
I20260619 08:18:56.928225 5974 raft_consensus.cc:2454] T f06e423a36e1409b93fbdcd54bbade31 P a3074c15b6234b4fa904343df8d5be42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcfbd0af911b4ba4ae77012eb6fb3286 in term 1.
I20260619 08:18:56.928316 6330 raft_consensus.cc:2781] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:18:56.928474 6330 raft_consensus.cc:686] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 1 LEADER]: Becoming Leader. State: Replica: fcfbd0af911b4ba4ae77012eb6fb3286, State: Running, Role: LEADER
I20260619 08:18:56.928542 6330 consensus_queue.cc:237] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:56.929361 5816 catalog_manager.cc:5495] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286 reported cstate change: term changed from 0 to 1, leader changed from <none> to fcfbd0af911b4ba4ae77012eb6fb3286 (127.4.151.130). New cstate: current_term: 1 leader_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } health_report { overall_health: UNKNOWN } } }
Jun 19 08:18:56 dist-test-slave-d3n8 krb5kdc[5772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857136, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:18:56.973187 6359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.966151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:45811 (local address 127.0.0.1:36820)
0619 08:18:56.966302 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:56.966312 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:56.966378 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:56.966513 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:56.966514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:56.966810 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:18:56.966861 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.966862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.967613 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.967614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.968140 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:56.968142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.968487 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.968488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:56.968537 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.969092 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:18:56.969097 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:18:56.970728 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:18:56.972222 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:56.972228 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:56.972230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:56.972421 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:56.972596 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:56.972596 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:56.972597 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:56.972651 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:56.972914 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:18:56.972920 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:18:56.972958 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 08:18:56.973014 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 08:18:56.973104 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260619 08:18:56.973245 5866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:56.966198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36820 (local address 127.4.151.190:45811)
0619 08:18:56.966263 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:56.966266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:56.966413 (+ 147us) server_negotiation.cc:407] Connection header received
0619 08:18:56.966436 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:56.966437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:56.966452 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:56.966492 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:56.967002 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.967511 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.968221 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:56.968402 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:56.968486 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:56.970822 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:56.970823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:56.970823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:56.970830 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:56.972142 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:56.972484 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:56.972485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:56.972486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:56.972538 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:56.972735 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:56.972736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:56.972736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:56.972851 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:56.972918 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:56.973052 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 08:18:56.973155 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":37}
I20260619 08:18:56.975487 5816 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:36820:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 08:18:56.975716 5816 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:18:56.977283 5954 tablet_service.cc:1467] Processing CreateTablet for tablet da90ffd8108d4502aaaf05c249d2a3a4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a1d2672974154a36a09f33b8d09feaee]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 08:18:56.977340 6234 tablet_service.cc:1467] Processing CreateTablet for tablet da90ffd8108d4502aaaf05c249d2a3a4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a1d2672974154a36a09f33b8d09feaee]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 08:18:56.977488 6234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da90ffd8108d4502aaaf05c249d2a3a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:18:56.977488 5954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da90ffd8108d4502aaaf05c249d2a3a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:18:56.977439 6094 tablet_service.cc:1467] Processing CreateTablet for tablet da90ffd8108d4502aaaf05c249d2a3a4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a1d2672974154a36a09f33b8d09feaee]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 08:18:56.977592 6094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da90ffd8108d4502aaaf05c249d2a3a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:18:56.979018 6325 tablet_bootstrap.cc:492] T da90ffd8108d4502aaaf05c249d2a3a4 P 2e1841f31d43409792c0c0eb2fc8d74e: Bootstrap starting.
I20260619 08:18:56.979259 6324 tablet_bootstrap.cc:492] T da90ffd8108d4502aaaf05c249d2a3a4 P fcfbd0af911b4ba4ae77012eb6fb3286: Bootstrap starting.
I20260619 08:18:56.979472 6325 tablet_bootstrap.cc:654] T da90ffd8108d4502aaaf05c249d2a3a4 P 2e1841f31d43409792c0c0eb2fc8d74e: Neither blocks nor log segments found. Creating new log.
I20260619 08:18:56.979686 6324 tablet_bootstrap.cc:654] T da90ffd8108d4502aaaf05c249d2a3a4 P fcfbd0af911b4ba4ae77012eb6fb3286: Neither blocks nor log segments found. Creating new log.
I20260619 08:18:56.979851 6323 tablet_bootstrap.cc:492] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42: Bootstrap starting.
I20260619 08:18:56.980139 6325 tablet_bootstrap.cc:492] T da90ffd8108d4502aaaf05c249d2a3a4 P 2e1841f31d43409792c0c0eb2fc8d74e: No bootstrap required, opened a new log
I20260619 08:18:56.980149 6323 tablet_bootstrap.cc:654] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42: Neither blocks nor log segments found. Creating new log.
I20260619 08:18:56.980185 6325 ts_tablet_manager.cc:1397] T da90ffd8108d4502aaaf05c249d2a3a4 P 2e1841f31d43409792c0c0eb2fc8d74e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:18:56.980345 6325 raft_consensus.cc:348] T da90ffd8108d4502aaaf05c249d2a3a4 P 2e1841f31d43409792c0c0eb2fc8d74e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:56.980436 6325 raft_consensus.cc:374] T da90ffd8108d4502aaaf05c249d2a3a4 P 2e1841f31d43409792c0c0eb2fc8d74e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:18:56.980463 6325 raft_consensus.cc:729] T da90ffd8108d4502aaaf05c249d2a3a4 P 2e1841f31d43409792c0c0eb2fc8d74e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e1841f31d43409792c0c0eb2fc8d74e, State: Initialized, Role: FOLLOWER
I20260619 08:18:56.980522 6325 consensus_queue.cc:260] T da90ffd8108d4502aaaf05c249d2a3a4 P 2e1841f31d43409792c0c0eb2fc8d74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:56.980638 6325 ts_tablet_manager.cc:1428] T da90ffd8108d4502aaaf05c249d2a3a4 P 2e1841f31d43409792c0c0eb2fc8d74e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 08:18:56.980710 6323 tablet_bootstrap.cc:492] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42: No bootstrap required, opened a new log
I20260619 08:18:56.980700 6331 tablet_replica.cc:439] T da90ffd8108d4502aaaf05c249d2a3a4 P 2e1841f31d43409792c0c0eb2fc8d74e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } } }
I20260619 08:18:56.980753 6323 ts_tablet_manager.cc:1397] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 08:18:56.980779 6331 tablet_replica.cc:442] T da90ffd8108d4502aaaf05c249d2a3a4 P 2e1841f31d43409792c0c0eb2fc8d74e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:18:56.980867 6323 raft_consensus.cc:348] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:56.980947 6323 raft_consensus.cc:374] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:18:56.980976 6323 raft_consensus.cc:729] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3074c15b6234b4fa904343df8d5be42, State: Initialized, Role: FOLLOWER
I20260619 08:18:56.981029 6323 consensus_queue.cc:260] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:56.981134 6323 ts_tablet_manager.cc:1428] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 08:18:56.981223 6333 tablet_replica.cc:439] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } } }
I20260619 08:18:56.981295 6333 tablet_replica.cc:442] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:18:56.981720 6324 tablet_bootstrap.cc:492] T da90ffd8108d4502aaaf05c249d2a3a4 P fcfbd0af911b4ba4ae77012eb6fb3286: No bootstrap required, opened a new log
I20260619 08:18:56.981775 6324 ts_tablet_manager.cc:1397] T da90ffd8108d4502aaaf05c249d2a3a4 P fcfbd0af911b4ba4ae77012eb6fb3286: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 08:18:56.981922 6324 raft_consensus.cc:348] T da90ffd8108d4502aaaf05c249d2a3a4 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:56.981977 6324 raft_consensus.cc:374] T da90ffd8108d4502aaaf05c249d2a3a4 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:18:56.981992 6324 raft_consensus.cc:729] T da90ffd8108d4502aaaf05c249d2a3a4 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcfbd0af911b4ba4ae77012eb6fb3286, State: Initialized, Role: FOLLOWER
I20260619 08:18:56.982059 6324 consensus_queue.cc:260] T da90ffd8108d4502aaaf05c249d2a3a4 P fcfbd0af911b4ba4ae77012eb6fb3286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:56.982148 6324 ts_tablet_manager.cc:1428] T da90ffd8108d4502aaaf05c249d2a3a4 P fcfbd0af911b4ba4ae77012eb6fb3286: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:56.982192 6330 tablet_replica.cc:439] T da90ffd8108d4502aaaf05c249d2a3a4 P fcfbd0af911b4ba4ae77012eb6fb3286: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } } }
I20260619 08:18:56.982256 6330 tablet_replica.cc:442] T da90ffd8108d4502aaaf05c249d2a3a4 P fcfbd0af911b4ba4ae77012eb6fb3286: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:18:57.025457 6333 raft_consensus.cc:482] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:18:57.025621 6333 raft_consensus.cc:504] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:57.025820 6333 leader_election.cc:288] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcfbd0af911b4ba4ae77012eb6fb3286 (127.4.151.130:35729), 2e1841f31d43409792c0c0eb2fc8d74e (127.4.151.131:36799)
I20260619 08:18:57.026168 6114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90ffd8108d4502aaaf05c249d2a3a4" candidate_uuid: "a3074c15b6234b4fa904343df8d5be42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" is_pre_election: true
I20260619 08:18:57.026216 6254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90ffd8108d4502aaaf05c249d2a3a4" candidate_uuid: "a3074c15b6234b4fa904343df8d5be42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" is_pre_election: true
I20260619 08:18:57.026288 6114 raft_consensus.cc:2454] T da90ffd8108d4502aaaf05c249d2a3a4 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3074c15b6234b4fa904343df8d5be42 in term 0.
I20260619 08:18:57.026328 6254 raft_consensus.cc:2454] T da90ffd8108d4502aaaf05c249d2a3a4 P 2e1841f31d43409792c0c0eb2fc8d74e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3074c15b6234b4fa904343df8d5be42 in term 0.
I20260619 08:18:57.026477 5907 leader_election.cc:302] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e1841f31d43409792c0c0eb2fc8d74e, a3074c15b6234b4fa904343df8d5be42; no voters:
I20260619 08:18:57.026607 6333 raft_consensus.cc:2781] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:18:57.026676 6333 raft_consensus.cc:482] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:18:57.026710 6333 raft_consensus.cc:3037] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:18:57.027514 6333 raft_consensus.cc:504] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:57.027661 6333 leader_election.cc:288] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42 [CANDIDATE]: Term 1 election: Requested vote from peers fcfbd0af911b4ba4ae77012eb6fb3286 (127.4.151.130:35729), 2e1841f31d43409792c0c0eb2fc8d74e (127.4.151.131:36799)
I20260619 08:18:57.027889 6254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90ffd8108d4502aaaf05c249d2a3a4" candidate_uuid: "a3074c15b6234b4fa904343df8d5be42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e1841f31d43409792c0c0eb2fc8d74e"
I20260619 08:18:57.027904 6114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90ffd8108d4502aaaf05c249d2a3a4" candidate_uuid: "a3074c15b6234b4fa904343df8d5be42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286"
I20260619 08:18:57.027981 6254 raft_consensus.cc:3037] T da90ffd8108d4502aaaf05c249d2a3a4 P 2e1841f31d43409792c0c0eb2fc8d74e [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:18:57.027981 6114 raft_consensus.cc:3037] T da90ffd8108d4502aaaf05c249d2a3a4 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:18:57.028743 6254 raft_consensus.cc:2454] T da90ffd8108d4502aaaf05c249d2a3a4 P 2e1841f31d43409792c0c0eb2fc8d74e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3074c15b6234b4fa904343df8d5be42 in term 1.
I20260619 08:18:57.028743 6114 raft_consensus.cc:2454] T da90ffd8108d4502aaaf05c249d2a3a4 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3074c15b6234b4fa904343df8d5be42 in term 1.
I20260619 08:18:57.028928 5907 leader_election.cc:302] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e1841f31d43409792c0c0eb2fc8d74e, a3074c15b6234b4fa904343df8d5be42; no voters:
I20260619 08:18:57.029106 6333 raft_consensus.cc:2781] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:18:57.029179 6333 raft_consensus.cc:686] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42 [term 1 LEADER]: Becoming Leader. State: Replica: a3074c15b6234b4fa904343df8d5be42, State: Running, Role: LEADER
I20260619 08:18:57.029241 6333 consensus_queue.cc:237] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } }
I20260619 08:18:57.029597 6329 tablet_replica.cc:439] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42: TxnStatusTablet state changed. Reason: New leader a3074c15b6234b4fa904343df8d5be42. Latest consensus state: current_term: 1 leader_uuid: "a3074c15b6234b4fa904343df8d5be42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } } }
I20260619 08:18:57.029690 6329 tablet_replica.cc:442] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42: This TxnStatusTablet replica's current role is: LEADER
I20260619 08:18:57.029842 6364 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 08:18:57.029937 6364 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 08:18:57.029938 5816 catalog_manager.cc:5495] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3074c15b6234b4fa904343df8d5be42 (127.4.151.129). New cstate: current_term: 1 leader_uuid: "a3074c15b6234b4fa904343df8d5be42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } health_report { overall_health: UNKNOWN } } }
I20260619 08:18:57.041882 6320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:57.039032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36052 (local address 127.4.151.129:41873)
0619 08:18:57.039092 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:57.039094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:57.039122 (+ 28us) server_negotiation.cc:407] Connection header received
0619 08:18:57.039203 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:57.039204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:57.039215 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:57.039287 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:18:57.039413 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:57.040162 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:57.040924 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:57.041224 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:57.041297 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:57.041598 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:18:57.041638 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:18:57.041690 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:57.041776 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 08:18:57.041799 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 08:18:57.041882 6359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:57.038921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:41873 (local address 127.0.0.1:36052)
0619 08:18:57.039070 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:57.039082 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:57.039139 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:57.039296 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:57.039296 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:57.039297 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:18:57.039353 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:57.039355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:57.040256 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:57.040257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:57.040843 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:57.040846 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:57.041347 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:57.041349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:57.041490 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:57.041507 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:18:57.041714 (+ 207us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:18:57.041725 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 08:18:57.041772 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 08:18:57.041820 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 08:18:57.043157 6254 raft_consensus.cc:1261] T da90ffd8108d4502aaaf05c249d2a3a4 P 2e1841f31d43409792c0c0eb2fc8d74e [term 1 FOLLOWER]: Refusing update from remote peer a3074c15b6234b4fa904343df8d5be42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:18:57.043159 6114 raft_consensus.cc:1261] T da90ffd8108d4502aaaf05c249d2a3a4 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 1 FOLLOWER]: Refusing update from remote peer a3074c15b6234b4fa904343df8d5be42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:18:57.043473 6329 consensus_queue.cc:1035] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:18:57.043624 6333 consensus_queue.cc:1035] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:18:57.044191 6331 tablet_replica.cc:439] T da90ffd8108d4502aaaf05c249d2a3a4 P 2e1841f31d43409792c0c0eb2fc8d74e: TxnStatusTablet state changed. Reason: New leader a3074c15b6234b4fa904343df8d5be42. Latest consensus state: current_term: 1 leader_uuid: "a3074c15b6234b4fa904343df8d5be42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } } }
I20260619 08:18:57.044292 6331 tablet_replica.cc:442] T da90ffd8108d4502aaaf05c249d2a3a4 P 2e1841f31d43409792c0c0eb2fc8d74e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:18:57.044301 6330 tablet_replica.cc:439] T da90ffd8108d4502aaaf05c249d2a3a4 P fcfbd0af911b4ba4ae77012eb6fb3286: TxnStatusTablet state changed. Reason: New leader a3074c15b6234b4fa904343df8d5be42. Latest consensus state: current_term: 1 leader_uuid: "a3074c15b6234b4fa904343df8d5be42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } } }
I20260619 08:18:57.044400 6330 tablet_replica.cc:442] T da90ffd8108d4502aaaf05c249d2a3a4 P fcfbd0af911b4ba4ae77012eb6fb3286: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:18:57.044734 6333 tablet_replica.cc:439] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3074c15b6234b4fa904343df8d5be42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } } }
I20260619 08:18:57.044819 6333 tablet_replica.cc:442] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42: This TxnStatusTablet replica's current role is: LEADER
I20260619 08:18:57.044884 6337 mvcc.cc:205] Tried to move back new op lower bound from 7298486833326825472 to 7298486833272102912. Current Snapshot: MvccSnapshot[applied={T|T < 7298486833326825472}]
I20260619 08:18:57.044998 6330 tablet_replica.cc:439] T da90ffd8108d4502aaaf05c249d2a3a4 P fcfbd0af911b4ba4ae77012eb6fb3286: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3074c15b6234b4fa904343df8d5be42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } } }
I20260619 08:18:57.045069 6330 tablet_replica.cc:442] T da90ffd8108d4502aaaf05c249d2a3a4 P fcfbd0af911b4ba4ae77012eb6fb3286: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:18:57.046208 6329 tablet_replica.cc:439] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3074c15b6234b4fa904343df8d5be42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } } }
I20260619 08:18:57.046288 6329 tablet_replica.cc:442] T da90ffd8108d4502aaaf05c249d2a3a4 P a3074c15b6234b4fa904343df8d5be42: This TxnStatusTablet replica's current role is: LEADER
I20260619 08:18:57.046721 6331 tablet_replica.cc:439] T da90ffd8108d4502aaaf05c249d2a3a4 P 2e1841f31d43409792c0c0eb2fc8d74e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3074c15b6234b4fa904343df8d5be42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcfbd0af911b4ba4ae77012eb6fb3286" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35729 } } peers { permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 } } peers { permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 } } }
I20260619 08:18:57.046830 6331 tablet_replica.cc:442] T da90ffd8108d4502aaaf05c249d2a3a4 P 2e1841f31d43409792c0c0eb2fc8d74e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:18:57.050599 6319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:57.047618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43598 (local address 127.4.151.130:35729)
0619 08:18:57.047662 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:57.047664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:57.047675 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:18:57.047755 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:57.047756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:57.047776 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:57.047843 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:18:57.048049 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:57.048863 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:57.049711 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:57.050011 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:57.050085 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:57.050189 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:18:57.050324 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:18:57.050402 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:57.050478 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 08:18:57.050498 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:18:57.056488 6320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:57.053650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:52387 (local address 127.4.151.129:41873)
0619 08:18:57.053697 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:57.053698 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:57.053707 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:18:57.053732 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:57.053733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:57.053743 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:57.053812 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:18:57.053996 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:57.054761 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:57.055715 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:57.055999 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:57.056069 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:57.056239 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:18:57.056274 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:18:57.056327 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:57.056390 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 08:18:57.056411 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 08:18:57.056694 6371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:57.053448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:41873 (local address 127.4.151.130:52387)
0619 08:18:57.053614 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:57.053627 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:57.053677 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:57.053874 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:57.053875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:57.053876 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:18:57.053934 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:57.053935 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:57.054927 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:57.054928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:57.055635 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:57.055637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:57.056090 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:57.056091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:57.056150 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:57.056165 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:18:57.056333 (+ 168us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:18:57.056334 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 08:18:57.056380 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 08:18:57.056415 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260619 08:18:57.059393 6254 raft_consensus.cc:1261] T f06e423a36e1409b93fbdcd54bbade31 P 2e1841f31d43409792c0c0eb2fc8d74e [term 1 FOLLOWER]: Refusing update from remote peer fcfbd0af911b4ba4ae77012eb6fb3286: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:18:57.059441 5974 raft_consensus.cc:1261] T f06e423a36e1409b93fbdcd54bbade31 P a3074c15b6234b4fa904343df8d5be42 [term 1 FOLLOWER]: Refusing update from remote peer fcfbd0af911b4ba4ae77012eb6fb3286: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:18:57.059688 6352 consensus_queue.cc:1035] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3074c15b6234b4fa904343df8d5be42" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:18:57.059827 6330 consensus_queue.cc:1035] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e1841f31d43409792c0c0eb2fc8d74e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 36799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:18:57.072396 6319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:57.065797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:56217 (local address 127.4.151.130:35729)
0619 08:18:57.065837 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:57.065839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:57.069648 (+ 3809us) server_negotiation.cc:407] Connection header received
0619 08:18:57.069777 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:57.069778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:57.069788 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:57.069842 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:18:57.070003 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:57.070535 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:57.071660 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:57.071841 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:57.071900 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:57.072068 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:18:57.072104 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:18:57.072169 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:57.072284 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 08:18:57.072308 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 08:18:57.072449 6377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:57.065741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:35729 (local address 127.4.151.129:56217)
0619 08:18:57.069562 (+ 3821us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:57.069585 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:57.069648 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:57.069887 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:57.069888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:57.069891 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:18:57.069941 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:57.069943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:57.070610 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:57.070611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:57.071575 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:57.071578 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:57.071913 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:57.071914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:57.071973 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:57.071990 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:18:57.072184 (+ 194us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:18:57.072186 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:18:57.072260 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 08:18:57.072316 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3783,"thread_start_us":74,"threads_started":1}
I20260619 08:18:57.081323 5816 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:18:57.081511 5816 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:18:57.082216 5816 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50cfc77216e3413a8673d9aff3cce300: Sending DeleteTablet for 3 replicas of tablet f06e423a36e1409b93fbdcd54bbade31
I20260619 08:18:57.082471 5954 tablet_service.cc:1514] Processing DeleteTablet for tablet f06e423a36e1409b93fbdcd54bbade31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:18:57 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:36024
I20260619 08:18:57.082530 6234 tablet_service.cc:1514] Processing DeleteTablet for tablet f06e423a36e1409b93fbdcd54bbade31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:18:57 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:54700
I20260619 08:18:57.082580 6346 tablet_replica.cc:330] T f06e423a36e1409b93fbdcd54bbade31 P a3074c15b6234b4fa904343df8d5be42: stopping tablet replica
I20260619 08:18:57.082612 6347 tablet_replica.cc:330] T f06e423a36e1409b93fbdcd54bbade31 P 2e1841f31d43409792c0c0eb2fc8d74e: stopping tablet replica
I20260619 08:18:57.082631 6346 raft_consensus.cc:2229] T f06e423a36e1409b93fbdcd54bbade31 P a3074c15b6234b4fa904343df8d5be42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:18:57.082652 6347 raft_consensus.cc:2229] T f06e423a36e1409b93fbdcd54bbade31 P 2e1841f31d43409792c0c0eb2fc8d74e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:18:57.082665 6346 raft_consensus.cc:2258] T f06e423a36e1409b93fbdcd54bbade31 P a3074c15b6234b4fa904343df8d5be42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:18:57.082686 6347 raft_consensus.cc:2258] T f06e423a36e1409b93fbdcd54bbade31 P 2e1841f31d43409792c0c0eb2fc8d74e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:18:57.083037 6346 ts_tablet_manager.cc:1905] T f06e423a36e1409b93fbdcd54bbade31 P a3074c15b6234b4fa904343df8d5be42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:18:57.083317 6347 ts_tablet_manager.cc:1905] T f06e423a36e1409b93fbdcd54bbade31 P 2e1841f31d43409792c0c0eb2fc8d74e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:18:57.084837 6346 ts_tablet_manager.cc:1918] T f06e423a36e1409b93fbdcd54bbade31 P a3074c15b6234b4fa904343df8d5be42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:18:57.084893 6346 log.cc:1196] T f06e423a36e1409b93fbdcd54bbade31 P a3074c15b6234b4fa904343df8d5be42: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-0/wal/wals/f06e423a36e1409b93fbdcd54bbade31
I20260619 08:18:57.085100 6346 ts_tablet_manager.cc:1939] T f06e423a36e1409b93fbdcd54bbade31 P a3074c15b6234b4fa904343df8d5be42: Deleting consensus metadata
I20260619 08:18:57.085604 6094 tablet_service.cc:1514] Processing DeleteTablet for tablet f06e423a36e1409b93fbdcd54bbade31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:18:57 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:43586
I20260619 08:18:57.085623 5801 catalog_manager.cc:4841] TS a3074c15b6234b4fa904343df8d5be42 (127.4.151.129:41873): tablet f06e423a36e1409b93fbdcd54bbade31 (table test-table [id=c813862968d54a68a2e5dd68cbf77a09]) successfully deleted
I20260619 08:18:57.085718 6345 tablet_replica.cc:330] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286: stopping tablet replica
I20260619 08:18:57.085814 6345 raft_consensus.cc:2229] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:18:57.085930 6345 raft_consensus.cc:2258] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:18:57.085958 6347 ts_tablet_manager.cc:1918] T f06e423a36e1409b93fbdcd54bbade31 P 2e1841f31d43409792c0c0eb2fc8d74e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:18:57.086050 6347 log.cc:1196] T f06e423a36e1409b93fbdcd54bbade31 P 2e1841f31d43409792c0c0eb2fc8d74e: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/f06e423a36e1409b93fbdcd54bbade31
I20260619 08:18:57.086248 6347 ts_tablet_manager.cc:1939] T f06e423a36e1409b93fbdcd54bbade31 P 2e1841f31d43409792c0c0eb2fc8d74e: Deleting consensus metadata
I20260619 08:18:57.086319 6345 ts_tablet_manager.cc:1905] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:18:57.086570 5803 catalog_manager.cc:4841] TS 2e1841f31d43409792c0c0eb2fc8d74e (127.4.151.131:36799): tablet f06e423a36e1409b93fbdcd54bbade31 (table test-table [id=c813862968d54a68a2e5dd68cbf77a09]) successfully deleted
I20260619 08:18:57.088001 6345 ts_tablet_manager.cc:1918] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:18:57.088073 6345 log.cc:1196] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/f06e423a36e1409b93fbdcd54bbade31
I20260619 08:18:57.088315 6345 ts_tablet_manager.cc:1939] T f06e423a36e1409b93fbdcd54bbade31 P fcfbd0af911b4ba4ae77012eb6fb3286: Deleting consensus metadata
I20260619 08:18:57.088691 5803 catalog_manager.cc:4841] TS fcfbd0af911b4ba4ae77012eb6fb3286 (127.4.151.130:35729): tablet f06e423a36e1409b93fbdcd54bbade31 (table test-table [id=c813862968d54a68a2e5dd68cbf77a09]) successfully deleted
Jun 19 08:18:57 dist-test-slave-d3n8 krb5kdc[5772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857136, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.151.129@KRBTEST.COM
I20260619 08:18:57.100163 6320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:57.086647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36054 (local address 127.4.151.129:41873)
0619 08:18:57.089495 (+ 2848us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:57.089498 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:57.089509 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:18:57.089534 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:57.089535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:57.089546 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:57.089610 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:57.090354 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:57.091180 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:57.092308 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:57.092644 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:57.092720 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:57.096608 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:57.096622 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:57.096622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:57.096632 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:57.098795 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:57.099259 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:57.099262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:57.099263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:57.099328 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:57.099550 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:57.099551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:57.099551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:57.099731 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:57.099877 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:57.099970 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 08:18:57.100054 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2816}
W20260619 08:18:57.100715 6014 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36054
I20260619 08:18:57.113957 5866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:57.105680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36830 (local address 127.4.151.190:45811)
0619 08:18:57.105734 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:57.105736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:57.105838 (+ 102us) server_negotiation.cc:407] Connection header received
0619 08:18:57.105912 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:57.105913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:57.105925 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:57.105985 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:57.106707 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:57.107566 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:57.108736 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:57.109062 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:57.109109 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:57.110747 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:57.110749 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:57.110749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:57.110758 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:57.112555 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:57.112996 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:57.112997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:57.112998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:57.113060 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:57.113290 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:57.113291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:57.113291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:57.113493 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:57.113594 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:57.113752 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 08:18:57.113843 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260619 08:18:57.114179 5816 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36830
I20260619 08:18:57.114921 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 5889
I20260619 08:18:57.122215 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 6029
I20260619 08:18:57.127873 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 6169
I20260619 08:18:57.132324 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 5785
2026-06-19T08:18:57Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2407 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:18:57 dist-test-slave-d3n8 krb5kdc[6386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:18:57 dist-test-slave-d3n8 krb5kdc[6386](info): set up 2 sockets
Jun 19 08:18:57 dist-test-slave-d3n8 krb5kdc[6386](info): commencing operation
krb5kdc: starting...
W20260619 08:18:58.189584 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:18:58 dist-test-slave-d3n8 krb5kdc[6386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:18:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:18:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:18:58.316675 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:45777
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:45777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:18:58.403381 6399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:18:58.403612 6399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:18:58.403648 6399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:18:58.404824 6399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:18:58.404892 6399 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 08:18:58.404917 6399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:18:58.404947 6399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:18:58.404973 6399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:18:58.404999 6399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:18:58.406543 6399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:45777
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:45777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:18:58.406817 6399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:18:58.407042 6399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:18:58.411950 6405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:58.412073 6407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:58.411945 6404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:18:58.412312 6399 server_base.cc:1034] running on GCE node
I20260619 08:18:58.412516 6399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:18:58.412804 6399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:18:58.414116 6399 hybrid_clock.cc:648] HybridClock initialized: now 1781857138414098 us; error 33 us; skew 500 ppm
Jun 19 08:18:58 dist-test-slave-d3n8 krb5kdc[6386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857138, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:18:58.416639 6399 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:18:58.417109 6399 webserver.cc:460] Webserver started at http://127.4.151.190:39447/ using document root <none> and password file <none>
I20260619 08:18:58.417353 6399 fs_manager.cc:359] Metadata directory not provided
I20260619 08:18:58.417452 6399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:18:58.417558 6399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:18:58.418567 6399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "7a725ef00170490ab4763e0cc7297652"
format_stamp: "Formatted at 2026-06-19 08:18:58 on dist-test-slave-d3n8"
server_key: "9c131ca0bf24c01f31677c24e6124f4f"
server_key_iv: "d6055e130ec7e752bb08d65324043189"
server_key_version: "encryptionkey@0"
I20260619 08:18:58.418890 6399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "7a725ef00170490ab4763e0cc7297652"
format_stamp: "Formatted at 2026-06-19 08:18:58 on dist-test-slave-d3n8"
server_key: "9c131ca0bf24c01f31677c24e6124f4f"
server_key_iv: "d6055e130ec7e752bb08d65324043189"
server_key_version: "encryptionkey@0"
I20260619 08:18:58.420218 6399 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:18:58.421002 6414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:58.421202 6399 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:18:58.421283 6399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "7a725ef00170490ab4763e0cc7297652"
format_stamp: "Formatted at 2026-06-19 08:18:58 on dist-test-slave-d3n8"
server_key: "9c131ca0bf24c01f31677c24e6124f4f"
server_key_iv: "d6055e130ec7e752bb08d65324043189"
server_key_version: "encryptionkey@0"
I20260619 08:18:58.421394 6399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:18:58.433326 6399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:18:58.435304 6399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:18:58.435415 6399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:18:58.440070 6399 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:45777
I20260619 08:18:58.440114 6476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:45777 every 8 connection(s)
I20260619 08:18:58.440480 6399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:18:58.441112 6477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:18:58.441665 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 6399
I20260619 08:18:58.441771 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:18:58.441941 4702 external_mini_cluster.cc:1428] Setting key b639368a950eea351b4d560ecc386565
I20260619 08:18:58.444136 6477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652: Bootstrap starting.
I20260619 08:18:58.444840 6477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652: Neither blocks nor log segments found. Creating new log.
I20260619 08:18:58.445148 6477 log.cc:824] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652: Log is configured to *not* fsync() on all Append() calls
I20260619 08:18:58.445873 6477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652: No bootstrap required, opened a new log
I20260619 08:18:58.447746 6477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a725ef00170490ab4763e0cc7297652" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 45777 } }
I20260619 08:18:58.447897 6477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:18:58.447922 6477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a725ef00170490ab4763e0cc7297652, State: Initialized, Role: FOLLOWER
I20260619 08:18:58.448061 6477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a725ef00170490ab4763e0cc7297652" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 45777 } }
I20260619 08:18:58.448140 6477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:18:58.448177 6477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:18:58.448225 6477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:18:58.448992 6477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a725ef00170490ab4763e0cc7297652" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 45777 } }
Jun 19 08:18:58 dist-test-slave-d3n8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:18:58.449110 6477 leader_election.cc:302] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a725ef00170490ab4763e0cc7297652; no voters:
I20260619 08:18:58.449285 6477 leader_election.cc:288] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:18:58.449329 6482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:18:58.449502 6482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652 [term 1 LEADER]: Becoming Leader. State: Replica: 7a725ef00170490ab4763e0cc7297652, State: Running, Role: LEADER
I20260619 08:18:58.449591 6477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:18:58.449669 6482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a725ef00170490ab4763e0cc7297652" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 45777 } }
I20260619 08:18:58.450197 6484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a725ef00170490ab4763e0cc7297652" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a725ef00170490ab4763e0cc7297652" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 45777 } } }
I20260619 08:18:58.450215 6483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a725ef00170490ab4763e0cc7297652. Latest consensus state: current_term: 1 leader_uuid: "7a725ef00170490ab4763e0cc7297652" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a725ef00170490ab4763e0cc7297652" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 45777 } } }
I20260619 08:18:58.450382 6483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652 [sys.catalog]: This master's current role is: LEADER
I20260619 08:18:58.450536 6484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652 [sys.catalog]: This master's current role is: LEADER
I20260619 08:18:58.450642 6491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:18:58.451221 6491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:18:58.452857 6491 catalog_manager.cc:1347] Generated new cluster ID: 5f068856c2044df5a10713d5478d7450
I20260619 08:18:58.452920 6491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:18:58.453573 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:58.442483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48396 (local address 127.4.151.190:45777)
0619 08:18:58.442669 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:58.442679 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:58.442714 (+ 35us) server_negotiation.cc:407] Connection header received
0619 08:18:58.443258 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:58.443266 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:58.443327 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:58.443479 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:58.444108 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.445515 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.446331 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.446761 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.446886 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:58.449629 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:58.449633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:58.449633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:58.449645 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:58.451804 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:58.452230 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:58.452233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:58.452233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:58.452335 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:58.452657 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:58.452658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:58.452658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:58.452967 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:58.453131 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:58.453202 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 08:18:58.453337 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260619 08:18:58.465759 6491 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:18:58.466501 6491 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:18:58.473618 6491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652: Generated new TSK 0
I20260619 08:18:58.473817 6491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.151.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
I20260619 08:18:58.509725 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:45777
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:18:58.597934 6503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:18:58.598143 6503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:18:58.598169 6503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:18:58.599352 6503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:18:58.599396 6503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:18:58.599417 6503 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 08:18:58.599467 6503 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 08:18:58.599493 6503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:18:58.600965 6503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.151.190:45777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:18:58.601233 6503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:18:58.601624 6503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:18:58.607582 6509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:58.607539 6511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:18:58.607621 6503 server_base.cc:1034] running on GCE node
W20260619 08:18:58.607543 6508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:18:58.607976 6503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:18:58.608297 6503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:18:58.609457 6503 hybrid_clock.cc:648] HybridClock initialized: now 1781857138609429 us; error 43 us; skew 500 ppm
Jun 19 08:18:58 dist-test-slave-d3n8 krb5kdc[6386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857138, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:18:58.612134 6503 init.cc:377] Logged in from keytab as kudu/127.4.151.129@KRBTEST.COM (short username kudu)
I20260619 08:18:58.612583 6503 webserver.cc:460] Webserver started at http://127.4.151.129:45321/ using document root <none> and password file <none>
I20260619 08:18:58.612823 6503 fs_manager.cc:359] Metadata directory not provided
I20260619 08:18:58.612869 6503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:18:58.613029 6503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:18:58.614162 6503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "34bcc72af5fa41369e350065f960e11a"
format_stamp: "Formatted at 2026-06-19 08:18:58 on dist-test-slave-d3n8"
server_key: "9661f1c0e6aeb2280cc2ae3183e378f5"
server_key_iv: "0de3cb49b5396bfa5bf2857f95ede96f"
server_key_version: "encryptionkey@0"
I20260619 08:18:58.614516 6503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "34bcc72af5fa41369e350065f960e11a"
format_stamp: "Formatted at 2026-06-19 08:18:58 on dist-test-slave-d3n8"
server_key: "9661f1c0e6aeb2280cc2ae3183e378f5"
server_key_iv: "0de3cb49b5396bfa5bf2857f95ede96f"
server_key_version: "encryptionkey@0"
I20260619 08:18:58.615847 6503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 08:18:58.616655 6518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:58.616842 6503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:18:58.616914 6503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "34bcc72af5fa41369e350065f960e11a"
format_stamp: "Formatted at 2026-06-19 08:18:58 on dist-test-slave-d3n8"
server_key: "9661f1c0e6aeb2280cc2ae3183e378f5"
server_key_iv: "0de3cb49b5396bfa5bf2857f95ede96f"
server_key_version: "encryptionkey@0"
I20260619 08:18:58.616998 6503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:18:58.632387 6503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:18:58.634714 6503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:18:58.634869 6503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:18:58.635377 6503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:18:58.635440 6503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:58.635474 6503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:18:58.635504 6503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 08:18:58 dist-test-slave-d3n8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857138, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:18:58.642961 6503 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:42561
I20260619 08:18:58.643039 6633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:42561 every 8 connection(s)
I20260619 08:18:58.643402 6503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
I20260619 08:18:58.645025 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:58.635884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:35139 (local address 127.4.151.190:45777)
0619 08:18:58.635965 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:58.635968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:58.636217 (+ 249us) server_negotiation.cc:407] Connection header received
0619 08:18:58.636999 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:58.637001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:58.637016 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:58.637117 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:58.638063 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.638628 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.639636 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.639880 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.639972 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:58.642254 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:58.642256 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:58.642256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:58.642265 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:58.643944 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:58.644294 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:58.644295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:58.644295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:58.644346 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:58.644516 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:58.644516 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:58.644517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:58.644647 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:58.644696 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:58.644897 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 08:18:58.644961 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 08:18:58.645215 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:58.635863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:45777 (local address 127.4.151.129:35139)
0619 08:18:58.636071 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:58.636121 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:58.636843 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:58.637208 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:58.637213 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:58.637647 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:18:58.637961 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:58.637968 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.638728 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.638729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:58.639541 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:58.639543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.639996 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.639998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:58.640158 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:58.640699 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:18:58.640707 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:18:58.642156 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:18:58.644014 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:58.644019 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:58.644021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:58.644229 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:58.644386 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:58.644387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:58.644387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:58.644471 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:58.644687 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:18:58.644689 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:18:58.644795 (+ 106us) client_negotiation.cc:769] Sending connection context
0619 08:18:58.644872 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 08:18:58.644989 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":97,"threads_started":1}
I20260619 08:18:58.645757 6634 heartbeater.cc:347] Connected to a master server at 127.4.151.190:45777
I20260619 08:18:58.645838 6634 heartbeater.cc:464] Registering TS with master...
I20260619 08:18:58.645897 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 6503
I20260619 08:18:58.645977 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:18:58.646198 6634 heartbeater.cc:511] Master 127.4.151.190:45777 requested a full tablet report, sending...
I20260619 08:18:58.646224 4702 external_mini_cluster.cc:1428] Setting key bc4bdbeacc84980226e8841ba9c952df
I20260619 08:18:58.646754 6430 ts_manager.cc:194] Registered new tserver with Master: 34bcc72af5fa41369e350065f960e11a (127.4.151.129:42561)
I20260619 08:18:58.647518 6430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.129@KRBTEST.COM'} at 127.4.151.129:35139
I20260619 08:18:58.652946 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:58.646866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:49003 (local address 127.4.151.190:45777)
0619 08:18:58.646912 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:58.646914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:58.647077 (+ 163us) server_negotiation.cc:407] Connection header received
0619 08:18:58.647124 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:58.647125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:58.647135 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:58.647191 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:58.647567 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.648108 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.648826 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.649114 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.649193 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:58.650264 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:58.650265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:58.650266 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:58.650273 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:58.651809 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:58.652164 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:58.652165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:58.652166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:58.652223 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:58.652410 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:58.652411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:58.652411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:58.652522 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:58.652593 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:58.652752 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 08:18:58.652828 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 08:18:58.652953 6640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:58.646869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:45777 (local address 127.4.151.129:49003)
0619 08:18:58.646992 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:58.647005 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:58.647070 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:58.647194 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:58.647195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:58.647418 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:18:58.647492 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:58.647494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.648203 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.648204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:58.648746 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:58.648747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.649195 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.649196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:58.649274 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:58.649809 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:18:58.649813 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:18:58.650158 (+ 345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:18:58.651905 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:58.651906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:58.651907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:58.652082 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:58.652285 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:58.652286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:58.652286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:58.652344 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:58.652585 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:18:58.652585 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:18:58.652635 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 08:18:58.652726 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 08:18:58.652830 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.4.151.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
I20260619 08:18:58.674026 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:45777
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:18:58.757809 6643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:18:58.758129 6643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:18:58.758159 6643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:18:58.759296 6643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:18:58.759334 6643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:18:58.759354 6643 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 08:18:58.759366 6643 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 08:18:58.759383 6643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:18:58.760802 6643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.151.190:45777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:18:58.760998 6643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:18:58.761206 6643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:18:58.766494 6649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:58.766501 6648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:58.766517 6651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:18:58.766736 6643 server_base.cc:1034] running on GCE node
I20260619 08:18:58.766961 6643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:18:58.767251 6643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:18:58.768471 6643 hybrid_clock.cc:648] HybridClock initialized: now 1781857138768457 us; error 28 us; skew 500 ppm
Jun 19 08:18:58 dist-test-slave-d3n8 krb5kdc[6386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857138, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:18:58.771242 6643 init.cc:377] Logged in from keytab as kudu/127.4.151.130@KRBTEST.COM (short username kudu)
I20260619 08:18:58.771759 6643 webserver.cc:460] Webserver started at http://127.4.151.130:43221/ using document root <none> and password file <none>
I20260619 08:18:58.772012 6643 fs_manager.cc:359] Metadata directory not provided
I20260619 08:18:58.772061 6643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:18:58.772229 6643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:18:58.773499 6643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "fae665b09029444180774fb93b894a07"
format_stamp: "Formatted at 2026-06-19 08:18:58 on dist-test-slave-d3n8"
server_key: "f9c8b78ad5aec0b0cac3733a57c4c73b"
server_key_iv: "3f471523a3fdba9c36a23d9f5c167246"
server_key_version: "encryptionkey@0"
I20260619 08:18:58.773980 6643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "fae665b09029444180774fb93b894a07"
format_stamp: "Formatted at 2026-06-19 08:18:58 on dist-test-slave-d3n8"
server_key: "f9c8b78ad5aec0b0cac3733a57c4c73b"
server_key_iv: "3f471523a3fdba9c36a23d9f5c167246"
server_key_version: "encryptionkey@0"
I20260619 08:18:58.775501 6643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:18:58.776427 6658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:58.776638 6643 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:18:58.776736 6643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "fae665b09029444180774fb93b894a07"
format_stamp: "Formatted at 2026-06-19 08:18:58 on dist-test-slave-d3n8"
server_key: "f9c8b78ad5aec0b0cac3733a57c4c73b"
server_key_iv: "3f471523a3fdba9c36a23d9f5c167246"
server_key_version: "encryptionkey@0"
I20260619 08:18:58.776815 6643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:18:58.787699 6643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:18:58.790207 6643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:18:58.790527 6643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:18:58.791188 6643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:18:58.791240 6643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:58.791272 6643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:18:58.791299 6643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 08:18:58 dist-test-slave-d3n8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857138, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:18:58.798848 6643 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:42513
I20260619 08:18:58.798957 6773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:42513 every 8 connection(s)
I20260619 08:18:58.799273 6643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
I20260619 08:18:58.800920 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:58.791682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:47149 (local address 127.4.151.190:45777)
0619 08:18:58.791738 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:58.791740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:58.791975 (+ 235us) server_negotiation.cc:407] Connection header received
0619 08:18:58.792526 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:58.792527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:58.792541 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:58.792590 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:58.793426 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.793968 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.794991 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.795191 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.795264 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:58.797978 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:58.797980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:58.797980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:58.797989 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:58.799723 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:58.800125 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:58.800126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:58.800127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:58.800186 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:58.800397 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:58.800397 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:58.800398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:58.800523 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:58.800577 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:58.800789 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 08:18:58.800857 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 08:18:58.801160 6667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:58.791679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:45777 (local address 127.4.151.130:47149)
0619 08:18:58.791858 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:58.791893 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:58.792418 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:58.792661 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:58.792664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:58.793021 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:18:58.793313 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:58.793319 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.794062 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.794063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:58.794884 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:58.794886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.795269 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.795270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:58.795399 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:58.795926 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:18:58.795937 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:18:58.797878 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:18:58.799818 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:58.799821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:58.799823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:58.800030 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:58.800249 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:58.800250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:58.800250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:58.800342 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:58.800567 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:18:58.800569 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:18:58.800676 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 08:18:58.800755 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 08:18:58.800872 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":96,"threads_started":1}
I20260619 08:18:58.801662 6774 heartbeater.cc:347] Connected to a master server at 127.4.151.190:45777
I20260619 08:18:58.801770 6774 heartbeater.cc:464] Registering TS with master...
I20260619 08:18:58.801973 6774 heartbeater.cc:511] Master 127.4.151.190:45777 requested a full tablet report, sending...
I20260619 08:18:58.802336 6430 ts_manager.cc:194] Registered new tserver with Master: fae665b09029444180774fb93b894a07 (127.4.151.130:42513)
I20260619 08:18:58.802721 6430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.130@KRBTEST.COM'} at 127.4.151.130:47149
I20260619 08:18:58.807775 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:58.802839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:47493 (local address 127.4.151.190:45777)
0619 08:18:58.802928 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:58.802930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:58.803036 (+ 106us) server_negotiation.cc:407] Connection header received
0619 08:18:58.803057 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:58.803057 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:58.803067 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:58.803119 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:58.803459 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.803936 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.804594 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.804779 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.804839 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:58.805705 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:58.805706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:58.805707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:58.805713 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:58.806878 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:58.807201 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:58.807202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:58.807203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:58.807250 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:58.807390 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:58.807390 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:58.807390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:58.807515 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:58.807562 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:58.807665 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 08:18:58.807719 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 08:18:58.807778 6780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:58.802751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:45777 (local address 127.4.151.130:47493)
0619 08:18:58.802947 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:58.802957 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:58.803000 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:58.803134 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:58.803135 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:58.803322 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:18:58.803392 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:58.803393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.804016 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.804017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:58.804503 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:58.804504 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.804860 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.804861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:58.804914 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:58.805281 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:18:58.805285 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:18:58.805631 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:18:58.806943 (+ 1312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:58.806943 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:58.806944 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:58.807139 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:58.807304 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:58.807305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:58.807305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:58.807353 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:58.807563 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:18:58.807563 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:18:58.807597 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 08:18:58.807642 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 08:18:58.807710 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":77,"threads_started":1}
I20260619 08:18:58.809219 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 6643
I20260619 08:18:58.809343 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:18:58.809587 4702 external_mini_cluster.cc:1428] Setting key d3e29da0ff84ea9ae0e959107deeed11
WARNING: no policy specified for kudu/127.4.151.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
I20260619 08:18:58.837172 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:45777
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:18:58.920171 6783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:18:58.920403 6783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:18:58.920445 6783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:18:58.921787 6783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:18:58.921849 6783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:18:58.921888 6783 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 08:18:58.921931 6783 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 08:18:58.921977 6783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:18:58.923503 6783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.151.190:45777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:18:58.923776 6783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:18:58.924007 6783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:18:58.929211 6791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:58.929219 6788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:18:58.929211 6789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:18:58.929226 6783 server_base.cc:1034] running on GCE node
I20260619 08:18:58.929742 6783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:18:58.929980 6783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:18:58.931131 6783 hybrid_clock.cc:648] HybridClock initialized: now 1781857138931115 us; error 29 us; skew 500 ppm
Jun 19 08:18:58 dist-test-slave-d3n8 krb5kdc[6386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857138, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:18:58.933740 6783 init.cc:377] Logged in from keytab as kudu/127.4.151.131@KRBTEST.COM (short username kudu)
I20260619 08:18:58.934302 6783 webserver.cc:460] Webserver started at http://127.4.151.131:40899/ using document root <none> and password file <none>
I20260619 08:18:58.934532 6783 fs_manager.cc:359] Metadata directory not provided
I20260619 08:18:58.934579 6783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:18:58.934707 6783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:18:58.935777 6783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "7bf65bdee5064e3a9073783f51458a67"
format_stamp: "Formatted at 2026-06-19 08:18:58 on dist-test-slave-d3n8"
server_key: "0615a22740e22b3bc16290801eb70015"
server_key_iv: "b3a8047cf3fb06ab08bd77836d93d74e"
server_key_version: "encryptionkey@0"
I20260619 08:18:58.936146 6783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "7bf65bdee5064e3a9073783f51458a67"
format_stamp: "Formatted at 2026-06-19 08:18:58 on dist-test-slave-d3n8"
server_key: "0615a22740e22b3bc16290801eb70015"
server_key_iv: "b3a8047cf3fb06ab08bd77836d93d74e"
server_key_version: "encryptionkey@0"
I20260619 08:18:58.937638 6783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:18:58.938506 6798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:58.938705 6783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:18:58.938810 6783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "7bf65bdee5064e3a9073783f51458a67"
format_stamp: "Formatted at 2026-06-19 08:18:58 on dist-test-slave-d3n8"
server_key: "0615a22740e22b3bc16290801eb70015"
server_key_iv: "b3a8047cf3fb06ab08bd77836d93d74e"
server_key_version: "encryptionkey@0"
I20260619 08:18:58.938979 6783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:18:58.955761 6783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:18:58.958184 6783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:18:58.958338 6783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:18:58.958839 6783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:18:58.958899 6783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:58.958933 6783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:18:58.958992 6783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:58.966408 6783 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:40785
I20260619 08:18:58.966486 6913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:40785 every 8 connection(s)
I20260619 08:18:58.966843 6783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
Jun 19 08:18:58 dist-test-slave-d3n8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857138, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:18:58.970191 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:58.959505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:49443 (local address 127.4.151.190:45777)
0619 08:18:58.959588 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:58.959591 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:58.959876 (+ 285us) server_negotiation.cc:407] Connection header received
0619 08:18:58.960735 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:58.960737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:58.960760 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:58.960938 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:58.961734 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.962579 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.964306 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.964524 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.964582 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:58.967510 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:58.967512 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:58.967512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:58.967520 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:58.968855 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:58.969255 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:58.969256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:58.969257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:58.969324 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:58.969626 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:58.969627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:58.969628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:58.969749 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:58.969822 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:58.970058 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 08:18:58.970127 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 08:18:58.970330 6807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:58.959475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:45777 (local address 127.4.151.131:49443)
0619 08:18:58.959711 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:58.959775 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:58.960501 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:58.960919 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:58.960924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:58.961318 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:18:58.961618 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:58.961625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.962690 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.962691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:58.964199 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:58.964203 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.964629 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.964630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:58.964824 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:58.965489 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:18:58.965501 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:18:58.967409 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:18:58.968944 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:58.968950 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:58.968952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:58.969173 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:58.969434 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:58.969435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:58.969436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:58.969545 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:58.969813 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:18:58.969817 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:18:58.969934 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 08:18:58.970010 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 08:18:58.970141 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":118,"threads_started":1}
I20260619 08:18:58.970782 6914 heartbeater.cc:347] Connected to a master server at 127.4.151.190:45777
I20260619 08:18:58.970880 6914 heartbeater.cc:464] Registering TS with master...
I20260619 08:18:58.971061 6914 heartbeater.cc:511] Master 127.4.151.190:45777 requested a full tablet report, sending...
I20260619 08:18:58.971412 6431 ts_manager.cc:194] Registered new tserver with Master: 7bf65bdee5064e3a9073783f51458a67 (127.4.151.131:40785)
I20260619 08:18:58.971791 6431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.131@KRBTEST.COM'} at 127.4.151.131:49443
I20260619 08:18:58.972494 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 6783
I20260619 08:18:58.972582 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:18:58.972786 4702 external_mini_cluster.cc:1428] Setting key 2c3f880d6ac80111eb48baaa349d2a3f
I20260619 08:18:58.973555 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:18:58.978096 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:58.971837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:39149 (local address 127.4.151.190:45777)
0619 08:18:58.971878 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:58.971880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:58.972066 (+ 186us) server_negotiation.cc:407] Connection header received
0619 08:18:58.972136 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:58.972137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:58.972146 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:58.972200 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:58.972546 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.973105 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.974064 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.974324 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.974409 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:58.975500 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:58.975502 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:58.975502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:58.975508 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:58.976791 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:58.977211 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:58.977212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:58.977212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:58.977298 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:58.977539 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:58.977540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:58.977540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:58.977639 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:58.977692 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:58.977903 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 08:18:58.978034 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 08:18:58.978113 6920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:58.971777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:45777 (local address 127.4.151.131:39149)
0619 08:18:58.972005 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:58.972017 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:58.972078 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:58.972206 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:58.972206 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:58.972395 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:18:58.972463 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:58.972464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.973202 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.973203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:58.973971 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:58.973973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.974412 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.974413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:58.974487 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:58.974937 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:18:58.974942 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:18:58.975401 (+ 459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:18:58.976878 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:58.976879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:58.976880 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:58.977118 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:58.977400 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:58.977401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:58.977402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:58.977478 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:58.977704 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:18:58.977705 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:18:58.977763 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 08:18:58.977852 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 08:18:58.977966 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":1,"thread_start_us":180,"threads_started":1}
Jun 19 08:18:58 dist-test-slave-d3n8 krb5kdc[6386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 08:18:58 dist-test-slave-d3n8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:18:58.997599 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:58.990267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48410 (local address 127.4.151.190:45777)
0619 08:18:58.990342 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:58.990344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:58.990353 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:18:58.990384 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:58.990385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:58.990394 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:58.990456 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:58.991044 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.991547 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.992413 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:58.992600 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:58.992656 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:58.995001 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:58.995002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:58.995003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:58.995010 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:58.996284 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:58.996642 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:58.996643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:58.996643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:58.996700 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:58.997024 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:58.997026 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:58.997026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:58.997144 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:58.997235 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:58.997369 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 08:18:58.997496 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 08:18:58.999181 6431 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:18:59.000901 6431 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:18:59.007936 6933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.003625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:40785 (local address 127.0.0.1:58966)
0619 08:18:59.004154 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:59.004163 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:59.004208 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:59.004368 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:59.004369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:59.004371 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:18:59.004451 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:59.004453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.005487 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.005488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:59.006669 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:59.006672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.007606 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.007607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:59.007685 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.007697 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 08:18:59.007771 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 08:18:59.007833 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":8,"thread_start_us":91,"threads_started":1}
I20260619 08:18:59.007957 6936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.003987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58966 (local address 127.4.151.131:40785)
0619 08:18:59.004153 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:59.004156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:59.004205 (+ 49us) server_negotiation.cc:407] Connection header received
0619 08:18:59.004270 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:59.004272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:59.004286 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:59.004363 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:18:59.004544 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.005362 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.006761 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.007507 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.007583 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.007639 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:59.007798 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 08:18:59.007842 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":73,"threads_started":1}
I20260619 08:18:59.008031 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.003727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37420 (local address 127.4.151.129:42561)
0619 08:18:59.004088 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:59.004093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:59.004106 (+ 13us) server_negotiation.cc:407] Connection header received
0619 08:18:59.004151 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:59.004153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:59.004180 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:59.004271 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:18:59.004513 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.005684 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.006800 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.007509 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.007581 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.007639 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:59.007886 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 08:18:59.007929 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":84,"threads_started":1}
I20260619 08:18:59.008155 6935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.003871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55348 (local address 127.4.151.130:42513)
0619 08:18:59.004373 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:59.004378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:59.004388 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:18:59.004426 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:59.004427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:59.004444 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:59.004515 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:18:59.004703 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.005858 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.007051 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.007581 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.007649 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.007705 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:59.007983 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 08:18:59.008038 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":94,"threads_started":1}
I20260619 08:18:59.007947 6931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.003625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:42513 (local address 127.0.0.1:55348)
0619 08:18:59.003804 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:59.003845 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:59.003912 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:59.004580 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:59.004581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:59.004583 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:18:59.004623 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:59.004624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.005963 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.005964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:59.006964 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:59.006966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.007736 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.007737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:59.007801 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.007803 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:18:59.007858 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 08:18:59.007894 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":61,"threads_started":1}
I20260619 08:18:59.008507 6932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.003626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:42561 (local address 127.0.0.1:37420)
0619 08:18:59.004033 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:59.004042 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:59.004082 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:59.004362 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:59.004365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:59.004367 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:18:59.004439 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:59.004441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.005786 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.005787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:59.006688 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:59.006691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.007676 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.007677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:59.007733 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.007736 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:18:59.007804 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 08:18:59.007846 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":4,"thread_start_us":101,"threads_started":1}
I20260619 08:18:59.009430 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 9ccb732256ae41efbc7dd1938f43fa2c (DEFAULT_TABLE table=test-table [id=11cc53abd7c345c3b258456c13fc68c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:18:59.009404 6848 tablet_service.cc:1467] Processing CreateTablet for tablet 9ccb732256ae41efbc7dd1938f43fa2c (DEFAULT_TABLE table=test-table [id=11cc53abd7c345c3b258456c13fc68c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:18:59.009452 6708 tablet_service.cc:1467] Processing CreateTablet for tablet 9ccb732256ae41efbc7dd1938f43fa2c (DEFAULT_TABLE table=test-table [id=11cc53abd7c345c3b258456c13fc68c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:18:59.009779 6708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ccb732256ae41efbc7dd1938f43fa2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:18:59.009784 6848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ccb732256ae41efbc7dd1938f43fa2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:18:59.009784 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ccb732256ae41efbc7dd1938f43fa2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:18:59.012504 6937 tablet_bootstrap.cc:492] T 9ccb732256ae41efbc7dd1938f43fa2c P fae665b09029444180774fb93b894a07: Bootstrap starting.
I20260619 08:18:59.013139 6938 tablet_bootstrap.cc:492] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67: Bootstrap starting.
I20260619 08:18:59.013185 6937 tablet_bootstrap.cc:654] T 9ccb732256ae41efbc7dd1938f43fa2c P fae665b09029444180774fb93b894a07: Neither blocks nor log segments found. Creating new log.
I20260619 08:18:59.013609 6937 log.cc:824] T 9ccb732256ae41efbc7dd1938f43fa2c P fae665b09029444180774fb93b894a07: Log is configured to *not* fsync() on all Append() calls
I20260619 08:18:59.014060 6939 tablet_bootstrap.cc:492] T 9ccb732256ae41efbc7dd1938f43fa2c P 34bcc72af5fa41369e350065f960e11a: Bootstrap starting.
I20260619 08:18:59.014389 6938 tablet_bootstrap.cc:654] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67: Neither blocks nor log segments found. Creating new log.
I20260619 08:18:59.014544 6937 tablet_bootstrap.cc:492] T 9ccb732256ae41efbc7dd1938f43fa2c P fae665b09029444180774fb93b894a07: No bootstrap required, opened a new log
I20260619 08:18:59.014635 6937 ts_tablet_manager.cc:1397] T 9ccb732256ae41efbc7dd1938f43fa2c P fae665b09029444180774fb93b894a07: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 08:18:59.014701 6939 tablet_bootstrap.cc:654] T 9ccb732256ae41efbc7dd1938f43fa2c P 34bcc72af5fa41369e350065f960e11a: Neither blocks nor log segments found. Creating new log.
I20260619 08:18:59.014729 6938 log.cc:824] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67: Log is configured to *not* fsync() on all Append() calls
I20260619 08:18:59.015024 6939 log.cc:824] T 9ccb732256ae41efbc7dd1938f43fa2c P 34bcc72af5fa41369e350065f960e11a: Log is configured to *not* fsync() on all Append() calls
I20260619 08:18:59.015561 6938 tablet_bootstrap.cc:492] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67: No bootstrap required, opened a new log
I20260619 08:18:59.015673 6938 ts_tablet_manager.cc:1397] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 08:18:59.015772 6939 tablet_bootstrap.cc:492] T 9ccb732256ae41efbc7dd1938f43fa2c P 34bcc72af5fa41369e350065f960e11a: No bootstrap required, opened a new log
I20260619 08:18:59.015838 6939 ts_tablet_manager.cc:1397] T 9ccb732256ae41efbc7dd1938f43fa2c P 34bcc72af5fa41369e350065f960e11a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:18:59.016301 6937 raft_consensus.cc:348] T 9ccb732256ae41efbc7dd1938f43fa2c P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.016438 6937 raft_consensus.cc:374] T 9ccb732256ae41efbc7dd1938f43fa2c P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:18:59.016466 6937 raft_consensus.cc:729] T 9ccb732256ae41efbc7dd1938f43fa2c P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fae665b09029444180774fb93b894a07, State: Initialized, Role: FOLLOWER
I20260619 08:18:59.016547 6937 consensus_queue.cc:260] T 9ccb732256ae41efbc7dd1938f43fa2c P fae665b09029444180774fb93b894a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.016793 6937 ts_tablet_manager.cc:1428] T 9ccb732256ae41efbc7dd1938f43fa2c P fae665b09029444180774fb93b894a07: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 08:18:59.016800 6774 heartbeater.cc:503] Master 127.4.151.190:45777 was elected leader, sending a full tablet report...
I20260619 08:18:59.018218 6939 raft_consensus.cc:348] T 9ccb732256ae41efbc7dd1938f43fa2c P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.018391 6939 raft_consensus.cc:374] T 9ccb732256ae41efbc7dd1938f43fa2c P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:18:59.018433 6939 raft_consensus.cc:729] T 9ccb732256ae41efbc7dd1938f43fa2c P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34bcc72af5fa41369e350065f960e11a, State: Initialized, Role: FOLLOWER
I20260619 08:18:59.018392 6938 raft_consensus.cc:348] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.018529 6938 raft_consensus.cc:374] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:18:59.018573 6938 raft_consensus.cc:729] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf65bdee5064e3a9073783f51458a67, State: Initialized, Role: FOLLOWER
I20260619 08:18:59.018553 6939 consensus_queue.cc:260] T 9ccb732256ae41efbc7dd1938f43fa2c P 34bcc72af5fa41369e350065f960e11a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.018680 6938 consensus_queue.cc:260] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.018828 6939 ts_tablet_manager.cc:1428] T 9ccb732256ae41efbc7dd1938f43fa2c P 34bcc72af5fa41369e350065f960e11a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:18:59.018850 6634 heartbeater.cc:503] Master 127.4.151.190:45777 was elected leader, sending a full tablet report...
I20260619 08:18:59.019017 6938 ts_tablet_manager.cc:1428] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:18:59.019137 6914 heartbeater.cc:503] Master 127.4.151.190:45777 was elected leader, sending a full tablet report...
W20260619 08:18:59.049886 6775 tablet.cc:2369] T 9ccb732256ae41efbc7dd1938f43fa2c P fae665b09029444180774fb93b894a07: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:18:59.144177 6635 tablet.cc:2369] T 9ccb732256ae41efbc7dd1938f43fa2c P 34bcc72af5fa41369e350065f960e11a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:18:59.147373 6945 raft_consensus.cc:482] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:18:59.147495 6945 raft_consensus.cc:504] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.147915 6945 leader_election.cc:288] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fae665b09029444180774fb93b894a07 (127.4.151.130:42513), 34bcc72af5fa41369e350065f960e11a (127.4.151.129:42561)
I20260619 08:18:59.151381 6807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.148177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:42513 (local address 127.4.151.131:47389)
0619 08:18:59.148238 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:59.148251 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:59.148298 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:59.148482 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:59.148483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:59.148485 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:18:59.148546 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:59.148548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.149405 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.149406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:59.150272 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:59.150275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.151117 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.151118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:59.151184 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.151188 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:18:59.151242 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 08:18:59.151274 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 08:18:59.151397 6935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.148227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:47389 (local address 127.4.151.130:42513)
0619 08:18:59.148284 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:59.148287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:59.148300 (+ 13us) server_negotiation.cc:407] Connection header received
0619 08:18:59.148397 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:59.148399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:59.148409 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:59.148468 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:18:59.148652 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.149299 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.150389 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.151002 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.151080 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.151094 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:59.151266 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 08:18:59.151323 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 08:18:59.151816 6946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.148179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:42561 (local address 127.4.151.131:34579)
0619 08:18:59.148404 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:59.148413 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:59.148460 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:59.148753 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:59.148754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:59.148756 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:18:59.148830 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:59.148832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.149598 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.149599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:59.150537 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:59.150539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.151337 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.151338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:59.151427 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.151431 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:18:59.151694 (+ 263us) client_negotiation.cc:240] Negotiation successful
0619 08:18:59.151743 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":54,"thread_start_us":113,"threads_started":1}
I20260619 08:18:59.151767 6728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ccb732256ae41efbc7dd1938f43fa2c" candidate_uuid: "7bf65bdee5064e3a9073783f51458a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fae665b09029444180774fb93b894a07" is_pre_election: true
I20260619 08:18:59.151831 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.148298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:34579 (local address 127.4.151.129:42561)
0619 08:18:59.148347 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:59.148349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:59.148634 (+ 285us) server_negotiation.cc:407] Connection header received
0619 08:18:59.148657 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:59.148658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:59.148677 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:59.148716 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:18:59.148910 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.149516 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.150667 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.151253 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.151330 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.151337 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:59.151724 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 08:18:59.151772 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 08:18:59.151939 6728 raft_consensus.cc:2454] T 9ccb732256ae41efbc7dd1938f43fa2c P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bf65bdee5064e3a9073783f51458a67 in term 0.
I20260619 08:18:59.152134 6588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ccb732256ae41efbc7dd1938f43fa2c" candidate_uuid: "7bf65bdee5064e3a9073783f51458a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34bcc72af5fa41369e350065f960e11a" is_pre_election: true
I20260619 08:18:59.152182 6802 leader_election.cc:302] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bf65bdee5064e3a9073783f51458a67, fae665b09029444180774fb93b894a07; no voters:
I20260619 08:18:59.152281 6588 raft_consensus.cc:2454] T 9ccb732256ae41efbc7dd1938f43fa2c P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bf65bdee5064e3a9073783f51458a67 in term 0.
I20260619 08:18:59.152321 6945 raft_consensus.cc:2781] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:18:59.152415 6945 raft_consensus.cc:482] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:18:59.152448 6945 raft_consensus.cc:3037] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:18:59.153301 6945 raft_consensus.cc:504] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.153515 6945 leader_election.cc:288] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67 [CANDIDATE]: Term 1 election: Requested vote from peers fae665b09029444180774fb93b894a07 (127.4.151.130:42513), 34bcc72af5fa41369e350065f960e11a (127.4.151.129:42561)
I20260619 08:18:59.153673 6588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ccb732256ae41efbc7dd1938f43fa2c" candidate_uuid: "7bf65bdee5064e3a9073783f51458a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34bcc72af5fa41369e350065f960e11a"
I20260619 08:18:59.153667 6728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ccb732256ae41efbc7dd1938f43fa2c" candidate_uuid: "7bf65bdee5064e3a9073783f51458a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fae665b09029444180774fb93b894a07"
I20260619 08:18:59.153743 6728 raft_consensus.cc:3037] T 9ccb732256ae41efbc7dd1938f43fa2c P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:18:59.153761 6588 raft_consensus.cc:3037] T 9ccb732256ae41efbc7dd1938f43fa2c P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:18:59.154511 6728 raft_consensus.cc:2454] T 9ccb732256ae41efbc7dd1938f43fa2c P fae665b09029444180774fb93b894a07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf65bdee5064e3a9073783f51458a67 in term 1.
I20260619 08:18:59.154518 6588 raft_consensus.cc:2454] T 9ccb732256ae41efbc7dd1938f43fa2c P 34bcc72af5fa41369e350065f960e11a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf65bdee5064e3a9073783f51458a67 in term 1.
I20260619 08:18:59.154971 6801 leader_election.cc:302] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34bcc72af5fa41369e350065f960e11a, 7bf65bdee5064e3a9073783f51458a67; no voters:
I20260619 08:18:59.155148 6945 raft_consensus.cc:2781] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:18:59.155321 6945 raft_consensus.cc:686] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67 [term 1 LEADER]: Becoming Leader. State: Replica: 7bf65bdee5064e3a9073783f51458a67, State: Running, Role: LEADER
I20260619 08:18:59.155416 6945 consensus_queue.cc:237] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.156229 6431 catalog_manager.cc:5495] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bf65bdee5064e3a9073783f51458a67 (127.4.151.131). New cstate: current_term: 1 leader_uuid: "7bf65bdee5064e3a9073783f51458a67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } health_report { overall_health: HEALTHY } } }
Jun 19 08:18:59 dist-test-slave-d3n8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857138, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:18:59.209355 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.202198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48422 (local address 127.4.151.190:45777)
0619 08:18:59.202291 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:59.202294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:59.202409 (+ 115us) server_negotiation.cc:407] Connection header received
0619 08:18:59.202468 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:59.202469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:59.202482 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:59.202581 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:18:59.203106 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.203598 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.204336 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.204520 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.204601 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.206721 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:18:59.206727 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:18:59.206727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:18:59.206734 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:18:59.208082 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:59.208584 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:59.208585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:59.208585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:59.208638 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:18:59.208815 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:18:59.208816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:18:59.208816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:18:59.208927 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:18:59.209001 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:59.209147 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 08:18:59.209244 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 08:18:59.209450 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.202137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:45777 (local address 127.0.0.1:48422)
0619 08:18:59.202311 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:59.202322 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:59.202405 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:59.202587 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:59.202588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:59.202975 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:18:59.203024 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:59.203027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.203684 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.203685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:59.204252 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:59.204254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.204599 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.204600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:59.204650 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.205213 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:18:59.205218 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:18:59.206627 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:18:59.208289 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:59.208296 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:59.208299 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:59.208498 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:59.208698 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:18:59.208699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:18:59.208699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:18:59.208756 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:18:59.208992 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:18:59.208999 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:18:59.209045 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 08:18:59.209125 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 08:18:59.209298 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":74,"thread_start_us":66,"threads_started":1}
I20260619 08:18:59.211964 6431 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:48422:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 08:18:59.212205 6431 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:18:59.213989 6848 tablet_service.cc:1467] Processing CreateTablet for tablet ffc7c9d3010641b7892960e226211f0e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0b3a9c73b536477eba33cb1df50db26d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 08:18:59.214025 6708 tablet_service.cc:1467] Processing CreateTablet for tablet ffc7c9d3010641b7892960e226211f0e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0b3a9c73b536477eba33cb1df50db26d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 08:18:59.214049 6568 tablet_service.cc:1467] Processing CreateTablet for tablet ffc7c9d3010641b7892960e226211f0e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0b3a9c73b536477eba33cb1df50db26d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 08:18:59.214170 6708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc7c9d3010641b7892960e226211f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:18:59.214178 6848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc7c9d3010641b7892960e226211f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:18:59.214188 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc7c9d3010641b7892960e226211f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:18:59.215693 6939 tablet_bootstrap.cc:492] T ffc7c9d3010641b7892960e226211f0e P 34bcc72af5fa41369e350065f960e11a: Bootstrap starting.
I20260619 08:18:59.215765 6938 tablet_bootstrap.cc:492] T ffc7c9d3010641b7892960e226211f0e P 7bf65bdee5064e3a9073783f51458a67: Bootstrap starting.
I20260619 08:18:59.215763 6937 tablet_bootstrap.cc:492] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07: Bootstrap starting.
I20260619 08:18:59.216213 6939 tablet_bootstrap.cc:654] T ffc7c9d3010641b7892960e226211f0e P 34bcc72af5fa41369e350065f960e11a: Neither blocks nor log segments found. Creating new log.
I20260619 08:18:59.216223 6938 tablet_bootstrap.cc:654] T ffc7c9d3010641b7892960e226211f0e P 7bf65bdee5064e3a9073783f51458a67: Neither blocks nor log segments found. Creating new log.
I20260619 08:18:59.216288 6937 tablet_bootstrap.cc:654] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07: Neither blocks nor log segments found. Creating new log.
I20260619 08:18:59.216918 6939 tablet_bootstrap.cc:492] T ffc7c9d3010641b7892960e226211f0e P 34bcc72af5fa41369e350065f960e11a: No bootstrap required, opened a new log
I20260619 08:18:59.216980 6939 ts_tablet_manager.cc:1397] T ffc7c9d3010641b7892960e226211f0e P 34bcc72af5fa41369e350065f960e11a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:18:59.217149 6939 raft_consensus.cc:348] T ffc7c9d3010641b7892960e226211f0e P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.217231 6939 raft_consensus.cc:374] T ffc7c9d3010641b7892960e226211f0e P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:18:59.217252 6939 raft_consensus.cc:729] T ffc7c9d3010641b7892960e226211f0e P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34bcc72af5fa41369e350065f960e11a, State: Initialized, Role: FOLLOWER
I20260619 08:18:59.217253 6937 tablet_bootstrap.cc:492] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07: No bootstrap required, opened a new log
I20260619 08:18:59.217294 6937 ts_tablet_manager.cc:1397] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 08:18:59.217307 6939 consensus_queue.cc:260] T ffc7c9d3010641b7892960e226211f0e P 34bcc72af5fa41369e350065f960e11a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.217483 6939 ts_tablet_manager.cc:1428] T ffc7c9d3010641b7892960e226211f0e P 34bcc72af5fa41369e350065f960e11a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 08:18:59.217473 6937 raft_consensus.cc:348] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.217556 6937 raft_consensus.cc:374] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:18:59.217526 6944 tablet_replica.cc:439] T ffc7c9d3010641b7892960e226211f0e P 34bcc72af5fa41369e350065f960e11a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } } }
I20260619 08:18:59.217588 6937 raft_consensus.cc:729] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fae665b09029444180774fb93b894a07, State: Initialized, Role: FOLLOWER
I20260619 08:18:59.217610 6938 tablet_bootstrap.cc:492] T ffc7c9d3010641b7892960e226211f0e P 7bf65bdee5064e3a9073783f51458a67: No bootstrap required, opened a new log
I20260619 08:18:59.217628 6944 tablet_replica.cc:442] T ffc7c9d3010641b7892960e226211f0e P 34bcc72af5fa41369e350065f960e11a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:18:59.217648 6937 consensus_queue.cc:260] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.217676 6938 ts_tablet_manager.cc:1397] T ffc7c9d3010641b7892960e226211f0e P 7bf65bdee5064e3a9073783f51458a67: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:18:59.217737 6937 ts_tablet_manager.cc:1428] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260619 08:18:59.217801 6915 tablet.cc:2369] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:18:59.217824 6938 raft_consensus.cc:348] T ffc7c9d3010641b7892960e226211f0e P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.217901 6938 raft_consensus.cc:374] T ffc7c9d3010641b7892960e226211f0e P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:18:59.217921 6938 raft_consensus.cc:729] T ffc7c9d3010641b7892960e226211f0e P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf65bdee5064e3a9073783f51458a67, State: Initialized, Role: FOLLOWER
I20260619 08:18:59.217968 6938 consensus_queue.cc:260] T ffc7c9d3010641b7892960e226211f0e P 7bf65bdee5064e3a9073783f51458a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.218065 6938 ts_tablet_manager.cc:1428] T ffc7c9d3010641b7892960e226211f0e P 7bf65bdee5064e3a9073783f51458a67: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:59.218076 6945 tablet_replica.cc:439] T ffc7c9d3010641b7892960e226211f0e P 7bf65bdee5064e3a9073783f51458a67: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } } }
I20260619 08:18:59.218196 6945 tablet_replica.cc:442] T ffc7c9d3010641b7892960e226211f0e P 7bf65bdee5064e3a9073783f51458a67: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:18:59.218189 6943 tablet_replica.cc:439] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } } }
I20260619 08:18:59.218245 6943 tablet_replica.cc:442] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:18:59.590827 6943 raft_consensus.cc:482] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:18:59.590939 6943 raft_consensus.cc:504] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.591359 6943 leader_election.cc:288] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34bcc72af5fa41369e350065f960e11a (127.4.151.129:42561), 7bf65bdee5064e3a9073783f51458a67 (127.4.151.131:40785)
I20260619 08:18:59.594400 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.591635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:33233 (local address 127.4.151.129:42561)
0619 08:18:59.591690 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:59.591693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:59.591783 (+ 90us) server_negotiation.cc:407] Connection header received
0619 08:18:59.591856 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:59.591857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:59.591868 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:59.591933 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:18:59.592115 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.592612 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.593666 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.594055 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.594141 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.594150 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:59.594297 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 08:18:59.594329 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 08:18:59.594413 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.591663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:56519 (local address 127.4.151.131:40785)
0619 08:18:59.591763 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:59.591765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:59.591775 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:18:59.591857 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:59.591858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:59.591873 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:59.591953 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:18:59.592128 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.592621 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.593678 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.594059 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.594142 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.594150 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:59.594292 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 08:18:59.594330 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":42,"threads_started":1}
I20260619 08:18:59.594429 6959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.591580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:40785 (local address 127.4.151.130:56519)
0619 08:18:59.591721 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:59.591731 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:59.591789 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:59.591955 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:59.591956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:59.591959 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:18:59.592053 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:59.592055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.592705 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.592706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:59.593593 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:59.593596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.594150 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.594150 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:59.594220 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.594223 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:18:59.594285 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 08:18:59.594327 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":53,"thread_start_us":65,"threads_started":1}
I20260619 08:18:59.594435 6958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.591567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:42561 (local address 127.4.151.130:33233)
0619 08:18:59.591708 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:59.591720 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:59.591795 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:59.591953 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:59.591954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:59.591956 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:18:59.592042 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:59.592044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.592697 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.592699 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:59.593581 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:59.593583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.594141 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.594142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:59.594212 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.594215 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:18:59.594279 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 08:18:59.594321 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":8,"thread_start_us":63,"threads_started":1}
I20260619 08:18:59.594841 6868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc7c9d3010641b7892960e226211f0e" candidate_uuid: "fae665b09029444180774fb93b894a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf65bdee5064e3a9073783f51458a67" is_pre_election: true
I20260619 08:18:59.594857 6588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc7c9d3010641b7892960e226211f0e" candidate_uuid: "fae665b09029444180774fb93b894a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34bcc72af5fa41369e350065f960e11a" is_pre_election: true
I20260619 08:18:59.594949 6588 raft_consensus.cc:2454] T ffc7c9d3010641b7892960e226211f0e P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fae665b09029444180774fb93b894a07 in term 0.
I20260619 08:18:59.594960 6868 raft_consensus.cc:2454] T ffc7c9d3010641b7892960e226211f0e P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fae665b09029444180774fb93b894a07 in term 0.
I20260619 08:18:59.595156 6661 leader_election.cc:302] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34bcc72af5fa41369e350065f960e11a, fae665b09029444180774fb93b894a07; no voters:
I20260619 08:18:59.595319 6943 raft_consensus.cc:2781] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:18:59.595423 6943 raft_consensus.cc:482] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:18:59.595458 6943 raft_consensus.cc:3037] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:18:59.596194 6943 raft_consensus.cc:504] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.596343 6943 leader_election.cc:288] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07 [CANDIDATE]: Term 1 election: Requested vote from peers 34bcc72af5fa41369e350065f960e11a (127.4.151.129:42561), 7bf65bdee5064e3a9073783f51458a67 (127.4.151.131:40785)
I20260619 08:18:59.596498 6868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc7c9d3010641b7892960e226211f0e" candidate_uuid: "fae665b09029444180774fb93b894a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf65bdee5064e3a9073783f51458a67"
I20260619 08:18:59.596520 6588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc7c9d3010641b7892960e226211f0e" candidate_uuid: "fae665b09029444180774fb93b894a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34bcc72af5fa41369e350065f960e11a"
I20260619 08:18:59.596585 6868 raft_consensus.cc:3037] T ffc7c9d3010641b7892960e226211f0e P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:18:59.596591 6588 raft_consensus.cc:3037] T ffc7c9d3010641b7892960e226211f0e P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:18:59.597203 6868 raft_consensus.cc:2454] T ffc7c9d3010641b7892960e226211f0e P 7bf65bdee5064e3a9073783f51458a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fae665b09029444180774fb93b894a07 in term 1.
I20260619 08:18:59.597229 6588 raft_consensus.cc:2454] T ffc7c9d3010641b7892960e226211f0e P 34bcc72af5fa41369e350065f960e11a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fae665b09029444180774fb93b894a07 in term 1.
I20260619 08:18:59.597393 6659 leader_election.cc:302] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bf65bdee5064e3a9073783f51458a67, fae665b09029444180774fb93b894a07; no voters:
I20260619 08:18:59.597500 6943 raft_consensus.cc:2781] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:18:59.597633 6943 raft_consensus.cc:686] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07 [term 1 LEADER]: Becoming Leader. State: Replica: fae665b09029444180774fb93b894a07, State: Running, Role: LEADER
I20260619 08:18:59.597736 6943 consensus_queue.cc:237] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.598162 6961 tablet_replica.cc:439] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07: TxnStatusTablet state changed. Reason: New leader fae665b09029444180774fb93b894a07. Latest consensus state: current_term: 1 leader_uuid: "fae665b09029444180774fb93b894a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } } }
I20260619 08:18:59.598243 6961 tablet_replica.cc:442] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07: This TxnStatusTablet replica's current role is: LEADER
I20260619 08:18:59.598475 6963 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 08:18:59.598585 6963 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 08:18:59.598493 6430 catalog_manager.cc:5495] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07 reported cstate change: term changed from 0 to 1, leader changed from <none> to fae665b09029444180774fb93b894a07 (127.4.151.130). New cstate: current_term: 1 leader_uuid: "fae665b09029444180774fb93b894a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } health_report { overall_health: UNKNOWN } } }
I20260619 08:18:59.600859 6945 consensus_queue.cc:1035] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:18:59.602825 6945 consensus_queue.cc:1035] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:18:59.665664 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.663011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:42513 (local address 127.0.0.1:54524)
0619 08:18:59.663105 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:59.663116 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:59.663175 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:59.663370 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:59.663371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:59.663372 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:18:59.663438 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:59.663441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.664082 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.664083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:59.664664 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:59.664665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.665039 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.665040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:59.665110 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.665134 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:18:59.665477 (+ 343us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:18:59.665486 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 08:18:59.665543 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 08:18:59.665594 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 08:18:59.665674 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.663073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54524 (local address 127.4.151.130:42513)
0619 08:18:59.663190 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:59.663192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:59.663211 (+ 19us) server_negotiation.cc:407] Connection header received
0619 08:18:59.663247 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:59.663248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:59.663267 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:59.663352 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:18:59.663521 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.664001 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.664758 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.664956 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.665037 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.665250 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:18:59.665347 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:18:59.665488 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:59.665549 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 08:18:59.665569 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
I20260619 08:18:59.667398 6588 raft_consensus.cc:1261] T ffc7c9d3010641b7892960e226211f0e P 34bcc72af5fa41369e350065f960e11a [term 1 FOLLOWER]: Refusing update from remote peer fae665b09029444180774fb93b894a07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:18:59.667496 6868 raft_consensus.cc:1261] T ffc7c9d3010641b7892960e226211f0e P 7bf65bdee5064e3a9073783f51458a67 [term 1 FOLLOWER]: Refusing update from remote peer fae665b09029444180774fb93b894a07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:18:59.667692 6961 consensus_queue.cc:1035] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:18:59.667804 6943 consensus_queue.cc:1035] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:18:59.668851 6944 tablet_replica.cc:439] T ffc7c9d3010641b7892960e226211f0e P 34bcc72af5fa41369e350065f960e11a: TxnStatusTablet state changed. Reason: New leader fae665b09029444180774fb93b894a07. Latest consensus state: current_term: 1 leader_uuid: "fae665b09029444180774fb93b894a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } } }
I20260619 08:18:59.668955 6944 tablet_replica.cc:442] T ffc7c9d3010641b7892960e226211f0e P 34bcc72af5fa41369e350065f960e11a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:18:59.669303 6945 tablet_replica.cc:439] T ffc7c9d3010641b7892960e226211f0e P 7bf65bdee5064e3a9073783f51458a67: TxnStatusTablet state changed. Reason: New leader fae665b09029444180774fb93b894a07. Latest consensus state: current_term: 1 leader_uuid: "fae665b09029444180774fb93b894a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } } }
I20260619 08:18:59.669438 6945 tablet_replica.cc:442] T ffc7c9d3010641b7892960e226211f0e P 7bf65bdee5064e3a9073783f51458a67: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:18:59.669788 6961 tablet_replica.cc:439] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fae665b09029444180774fb93b894a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } } }
I20260619 08:18:59.669876 6961 tablet_replica.cc:442] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07: This TxnStatusTablet replica's current role is: LEADER
I20260619 08:18:59.670423 6961 tablet_replica.cc:439] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fae665b09029444180774fb93b894a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } } }
I20260619 08:18:59.670502 6961 tablet_replica.cc:442] T ffc7c9d3010641b7892960e226211f0e P fae665b09029444180774fb93b894a07: This TxnStatusTablet replica's current role is: LEADER
I20260619 08:18:59.670754 6966 mvcc.cc:205] Tried to move back new op lower bound from 7298486844075261952 to 7298486843792850944. Current Snapshot: MvccSnapshot[applied={T|T < 7298486844075261952}]
I20260619 08:18:59.670843 6945 tablet_replica.cc:439] T ffc7c9d3010641b7892960e226211f0e P 7bf65bdee5064e3a9073783f51458a67: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fae665b09029444180774fb93b894a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } } }
I20260619 08:18:59.670909 6945 tablet_replica.cc:442] T ffc7c9d3010641b7892960e226211f0e P 7bf65bdee5064e3a9073783f51458a67: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:18:59.670950 6944 tablet_replica.cc:439] T ffc7c9d3010641b7892960e226211f0e P 34bcc72af5fa41369e350065f960e11a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fae665b09029444180774fb93b894a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } } }
I20260619 08:18:59.671036 6944 tablet_replica.cc:442] T ffc7c9d3010641b7892960e226211f0e P 34bcc72af5fa41369e350065f960e11a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:18:59.671167 6967 mvcc.cc:205] Tried to move back new op lower bound from 7298486844075261952 to 7298486843792850944. Current Snapshot: MvccSnapshot[applied={T|T < 7298486844075261952}]
I20260619 08:18:59.675130 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.672381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35070 (local address 127.4.151.131:40785)
0619 08:18:59.672430 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:59.672432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:59.672624 (+ 192us) server_negotiation.cc:407] Connection header received
0619 08:18:59.672699 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:59.672700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:59.672716 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:59.672783 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:18:59.672953 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.673494 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.674288 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.674515 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.674599 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.674776 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:18:59.674875 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:18:59.675003 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:59.675022 (+ 19us) server_negotiation.cc:299] Negotiation successful
0619 08:18:59.675052 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 08:18:59.680141 6981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.677513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:42513 (local address 127.4.151.131:47477)
0619 08:18:59.677667 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:59.677678 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:59.677754 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:59.677889 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:59.677890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:59.677892 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:18:59.678019 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:59.678021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.678664 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.678665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:59.679304 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:59.679306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.679700 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.679701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:59.679769 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.679782 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:18:59.679980 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:18:59.679981 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 08:18:59.680026 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 08:18:59.680067 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260619 08:18:59.680128 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.677578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:47477 (local address 127.4.151.130:42513)
0619 08:18:59.677620 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:59.677622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:59.677740 (+ 118us) server_negotiation.cc:407] Connection header received
0619 08:18:59.677805 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:59.677806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:59.677818 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:59.677885 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:18:59.678091 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.678577 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.679379 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.679589 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.679674 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.679892 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:18:59.679925 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:18:59.679974 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:59.680028 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 08:18:59.680048 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:18:59.690636 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.687638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:40527 (local address 127.4.151.131:40785)
0619 08:18:59.687696 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:59.687698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:59.687733 (+ 35us) server_negotiation.cc:407] Connection header received
0619 08:18:59.687800 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:59.687800 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:59.687812 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:59.687869 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:18:59.688108 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.688850 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.689923 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.690140 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.690215 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.690381 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:18:59.690415 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:18:59.690475 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:59.690528 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 08:18:59.690551 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 08:18:59.690666 6984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.687493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:40785 (local address 127.4.151.130:40527)
0619 08:18:59.687674 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:59.687688 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:59.687751 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:59.687871 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:59.687872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:59.687873 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:18:59.688035 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:59.688036 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.688926 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.688927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:59.689834 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:59.689837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.690211 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.690212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:59.690275 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.690292 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:18:59.690477 (+ 185us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:18:59.690479 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:18:59.690525 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 08:18:59.690567 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
I20260619 08:18:59.701020 6430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:18:59.701175 6430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:18:59.702154 6430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652: Sending DeleteTablet for 3 replicas of tablet 9ccb732256ae41efbc7dd1938f43fa2c
I20260619 08:18:59.702510 6708 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ccb732256ae41efbc7dd1938f43fa2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:18:59 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:55348
I20260619 08:18:59.702652 6568 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ccb732256ae41efbc7dd1938f43fa2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:18:59 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:37420
I20260619 08:18:59.703053 6848 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ccb732256ae41efbc7dd1938f43fa2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:18:59 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:58966
I20260619 08:18:59.703313 6430 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:18:59.703516 6430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:18:59.703766 6989 tablet_replica.cc:330] T 9ccb732256ae41efbc7dd1938f43fa2c P 34bcc72af5fa41369e350065f960e11a: stopping tablet replica
I20260619 08:18:59.705080 6568 tablet_service.cc:1467] Processing CreateTablet for tablet c68cc198778e4592a2d59e2439e71d04 (DEFAULT_TABLE table=test-table [id=1ce2cb6929f14d2195868ec9d6f41af2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:18:59.705240 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c68cc198778e4592a2d59e2439e71d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:18:59.705508 6990 tablet_replica.cc:330] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67: stopping tablet replica
I20260619 08:18:59.705687 6990 raft_consensus.cc:2229] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:18:59.705859 6990 raft_consensus.cc:2258] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:18:59.706950 6939 tablet_bootstrap.cc:492] T c68cc198778e4592a2d59e2439e71d04 P 34bcc72af5fa41369e350065f960e11a: Bootstrap starting.
I20260619 08:18:59.707310 6708 tablet_service.cc:1467] Processing CreateTablet for tablet c68cc198778e4592a2d59e2439e71d04 (DEFAULT_TABLE table=test-table [id=1ce2cb6929f14d2195868ec9d6f41af2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:18:59.707413 6939 tablet_bootstrap.cc:654] T c68cc198778e4592a2d59e2439e71d04 P 34bcc72af5fa41369e350065f960e11a: Neither blocks nor log segments found. Creating new log.
I20260619 08:18:59.707428 6708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c68cc198778e4592a2d59e2439e71d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:18:59.708166 6939 tablet_bootstrap.cc:492] T c68cc198778e4592a2d59e2439e71d04 P 34bcc72af5fa41369e350065f960e11a: No bootstrap required, opened a new log
I20260619 08:18:59.708218 6939 ts_tablet_manager.cc:1397] T c68cc198778e4592a2d59e2439e71d04 P 34bcc72af5fa41369e350065f960e11a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:18:59.708374 6939 raft_consensus.cc:348] T c68cc198778e4592a2d59e2439e71d04 P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.708453 6939 raft_consensus.cc:374] T c68cc198778e4592a2d59e2439e71d04 P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:18:59.708474 6939 raft_consensus.cc:729] T c68cc198778e4592a2d59e2439e71d04 P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34bcc72af5fa41369e350065f960e11a, State: Initialized, Role: FOLLOWER
I20260619 08:18:59.708528 6939 consensus_queue.cc:260] T c68cc198778e4592a2d59e2439e71d04 P 34bcc72af5fa41369e350065f960e11a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.708612 6939 ts_tablet_manager.cc:1428] T c68cc198778e4592a2d59e2439e71d04 P 34bcc72af5fa41369e350065f960e11a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:59.708842 6937 tablet_bootstrap.cc:492] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07: Bootstrap starting.
I20260619 08:18:59.709282 6937 tablet_bootstrap.cc:654] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07: Neither blocks nor log segments found. Creating new log.
I20260619 08:18:59.709918 6848 tablet_service.cc:1467] Processing CreateTablet for tablet c68cc198778e4592a2d59e2439e71d04 (DEFAULT_TABLE table=test-table [id=1ce2cb6929f14d2195868ec9d6f41af2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:18:59.710039 6848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c68cc198778e4592a2d59e2439e71d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:18:59.710115 6988 tablet_replica.cc:330] T 9ccb732256ae41efbc7dd1938f43fa2c P fae665b09029444180774fb93b894a07: stopping tablet replica
I20260619 08:18:59.710225 6988 raft_consensus.cc:2229] T 9ccb732256ae41efbc7dd1938f43fa2c P fae665b09029444180774fb93b894a07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:18:59.710291 6988 raft_consensus.cc:2258] T 9ccb732256ae41efbc7dd1938f43fa2c P fae665b09029444180774fb93b894a07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:18:59.710659 6990 ts_tablet_manager.cc:1905] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:18:59.711460 6938 tablet_bootstrap.cc:492] T c68cc198778e4592a2d59e2439e71d04 P 7bf65bdee5064e3a9073783f51458a67: Bootstrap starting.
I20260619 08:18:59.703902 6989 raft_consensus.cc:2229] T 9ccb732256ae41efbc7dd1938f43fa2c P 34bcc72af5fa41369e350065f960e11a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:18:59.711869 6938 tablet_bootstrap.cc:654] T c68cc198778e4592a2d59e2439e71d04 P 7bf65bdee5064e3a9073783f51458a67: Neither blocks nor log segments found. Creating new log.
I20260619 08:18:59.711879 6989 raft_consensus.cc:2258] T 9ccb732256ae41efbc7dd1938f43fa2c P 34bcc72af5fa41369e350065f960e11a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:18:59.712242 6988 ts_tablet_manager.cc:1905] T 9ccb732256ae41efbc7dd1938f43fa2c P fae665b09029444180774fb93b894a07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:18:59.712738 6938 tablet_bootstrap.cc:492] T c68cc198778e4592a2d59e2439e71d04 P 7bf65bdee5064e3a9073783f51458a67: No bootstrap required, opened a new log
I20260619 08:18:59.712793 6938 ts_tablet_manager.cc:1397] T c68cc198778e4592a2d59e2439e71d04 P 7bf65bdee5064e3a9073783f51458a67: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:18:59.712937 6938 raft_consensus.cc:348] T c68cc198778e4592a2d59e2439e71d04 P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.713016 6938 raft_consensus.cc:374] T c68cc198778e4592a2d59e2439e71d04 P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:18:59.713037 6938 raft_consensus.cc:729] T c68cc198778e4592a2d59e2439e71d04 P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf65bdee5064e3a9073783f51458a67, State: Initialized, Role: FOLLOWER
I20260619 08:18:59.713081 6938 consensus_queue.cc:260] T c68cc198778e4592a2d59e2439e71d04 P 7bf65bdee5064e3a9073783f51458a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.713208 6938 ts_tablet_manager.cc:1428] T c68cc198778e4592a2d59e2439e71d04 P 7bf65bdee5064e3a9073783f51458a67: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 08:18:59.714233 6989 ts_tablet_manager.cc:1905] T 9ccb732256ae41efbc7dd1938f43fa2c P 34bcc72af5fa41369e350065f960e11a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:18:59.715867 6989 ts_tablet_manager.cc:1918] T 9ccb732256ae41efbc7dd1938f43fa2c P 34bcc72af5fa41369e350065f960e11a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:18:59.715970 6989 log.cc:1196] T 9ccb732256ae41efbc7dd1938f43fa2c P 34bcc72af5fa41369e350065f960e11a: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-0/wal/wals/9ccb732256ae41efbc7dd1938f43fa2c
I20260619 08:18:59.716254 6989 ts_tablet_manager.cc:1939] T 9ccb732256ae41efbc7dd1938f43fa2c P 34bcc72af5fa41369e350065f960e11a: Deleting consensus metadata
I20260619 08:18:59.717338 6417 catalog_manager.cc:4841] TS 34bcc72af5fa41369e350065f960e11a (127.4.151.129:42561): tablet 9ccb732256ae41efbc7dd1938f43fa2c (table test-table [id=11cc53abd7c345c3b258456c13fc68c2]) successfully deleted
I20260619 08:18:59.717617 6988 ts_tablet_manager.cc:1918] T 9ccb732256ae41efbc7dd1938f43fa2c P fae665b09029444180774fb93b894a07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:18:59.717830 6990 ts_tablet_manager.cc:1918] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:18:59.717913 6990 log.cc:1196] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/9ccb732256ae41efbc7dd1938f43fa2c
I20260619 08:18:59.718009 6988 log.cc:1196] T 9ccb732256ae41efbc7dd1938f43fa2c P fae665b09029444180774fb93b894a07: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/9ccb732256ae41efbc7dd1938f43fa2c
I20260619 08:18:59.718189 6990 ts_tablet_manager.cc:1939] T 9ccb732256ae41efbc7dd1938f43fa2c P 7bf65bdee5064e3a9073783f51458a67: Deleting consensus metadata
I20260619 08:18:59.718436 6988 ts_tablet_manager.cc:1939] T 9ccb732256ae41efbc7dd1938f43fa2c P fae665b09029444180774fb93b894a07: Deleting consensus metadata
I20260619 08:18:59.718461 6937 tablet_bootstrap.cc:492] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07: No bootstrap required, opened a new log
I20260619 08:18:59.718560 6937 ts_tablet_manager.cc:1397] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.001s
I20260619 08:18:59.718716 6937 raft_consensus.cc:348] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.718797 6937 raft_consensus.cc:374] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:18:59.718822 6937 raft_consensus.cc:729] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fae665b09029444180774fb93b894a07, State: Initialized, Role: FOLLOWER
I20260619 08:18:59.718868 6937 consensus_queue.cc:260] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.718961 6937 ts_tablet_manager.cc:1428] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:18:59.719282 6418 catalog_manager.cc:4841] TS fae665b09029444180774fb93b894a07 (127.4.151.130:42513): tablet 9ccb732256ae41efbc7dd1938f43fa2c (table test-table [id=11cc53abd7c345c3b258456c13fc68c2]) successfully deleted
I20260619 08:18:59.719717 6415 catalog_manager.cc:4841] TS 7bf65bdee5064e3a9073783f51458a67 (127.4.151.131:40785): tablet 9ccb732256ae41efbc7dd1938f43fa2c (table test-table [id=11cc53abd7c345c3b258456c13fc68c2]) successfully deleted
I20260619 08:18:59.913726 6943 raft_consensus.cc:482] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:18:59.913940 6943 raft_consensus.cc:504] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.914132 6943 leader_election.cc:288] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34bcc72af5fa41369e350065f960e11a (127.4.151.129:42561), 7bf65bdee5064e3a9073783f51458a67 (127.4.151.131:40785)
I20260619 08:18:59.914345 6588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c68cc198778e4592a2d59e2439e71d04" candidate_uuid: "fae665b09029444180774fb93b894a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34bcc72af5fa41369e350065f960e11a" is_pre_election: true
I20260619 08:18:59.914463 6588 raft_consensus.cc:2454] T c68cc198778e4592a2d59e2439e71d04 P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fae665b09029444180774fb93b894a07 in term 0.
I20260619 08:18:59.914388 6868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c68cc198778e4592a2d59e2439e71d04" candidate_uuid: "fae665b09029444180774fb93b894a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf65bdee5064e3a9073783f51458a67" is_pre_election: true
I20260619 08:18:59.914532 6868 raft_consensus.cc:2454] T c68cc198778e4592a2d59e2439e71d04 P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fae665b09029444180774fb93b894a07 in term 0.
I20260619 08:18:59.914698 6659 leader_election.cc:302] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bf65bdee5064e3a9073783f51458a67, fae665b09029444180774fb93b894a07; no voters:
I20260619 08:18:59.914824 6943 raft_consensus.cc:2781] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:18:59.914880 6943 raft_consensus.cc:482] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:18:59.914901 6943 raft_consensus.cc:3037] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:18:59.915592 6943 raft_consensus.cc:504] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.915726 6943 leader_election.cc:288] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07 [CANDIDATE]: Term 1 election: Requested vote from peers 34bcc72af5fa41369e350065f960e11a (127.4.151.129:42561), 7bf65bdee5064e3a9073783f51458a67 (127.4.151.131:40785)
I20260619 08:18:59.915889 6588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c68cc198778e4592a2d59e2439e71d04" candidate_uuid: "fae665b09029444180774fb93b894a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34bcc72af5fa41369e350065f960e11a"
I20260619 08:18:59.915977 6588 raft_consensus.cc:3037] T c68cc198778e4592a2d59e2439e71d04 P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:18:59.915973 6868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c68cc198778e4592a2d59e2439e71d04" candidate_uuid: "fae665b09029444180774fb93b894a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf65bdee5064e3a9073783f51458a67"
I20260619 08:18:59.916072 6868 raft_consensus.cc:3037] T c68cc198778e4592a2d59e2439e71d04 P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:18:59.916637 6588 raft_consensus.cc:2454] T c68cc198778e4592a2d59e2439e71d04 P 34bcc72af5fa41369e350065f960e11a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fae665b09029444180774fb93b894a07 in term 1.
I20260619 08:18:59.916808 6661 leader_election.cc:302] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34bcc72af5fa41369e350065f960e11a, fae665b09029444180774fb93b894a07; no voters:
I20260619 08:18:59.916913 6943 raft_consensus.cc:2781] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:18:59.916956 6943 raft_consensus.cc:686] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07 [term 1 LEADER]: Becoming Leader. State: Replica: fae665b09029444180774fb93b894a07, State: Running, Role: LEADER
I20260619 08:18:59.917011 6943 consensus_queue.cc:237] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:18:59.917083 6868 raft_consensus.cc:2454] T c68cc198778e4592a2d59e2439e71d04 P 7bf65bdee5064e3a9073783f51458a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fae665b09029444180774fb93b894a07 in term 1.
I20260619 08:18:59.917724 6430 catalog_manager.cc:5495] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07 reported cstate change: term changed from 0 to 1, leader changed from <none> to fae665b09029444180774fb93b894a07 (127.4.151.130). New cstate: current_term: 1 leader_uuid: "fae665b09029444180774fb93b894a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } health_report { overall_health: UNKNOWN } } }
I20260619 08:18:59.941324 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.939135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54534 (local address 127.4.151.130:42513)
0619 08:18:59.939176 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:59.939178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:59.939213 (+ 35us) server_negotiation.cc:407] Connection header received
0619 08:18:59.939269 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:59.939270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:59.939300 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:59.939347 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:18:59.939462 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.939942 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.940661 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.940839 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.940899 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.941007 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:18:59.941032 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:18:59.941070 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:59.941196 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 08:18:59.941220 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 08:18:59.945807 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.943507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:49695 (local address 127.4.151.130:42513)
0619 08:18:59.943544 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 08:18:59.943546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:18:59.943592 (+ 46us) server_negotiation.cc:407] Connection header received
0619 08:18:59.943677 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:18:59.943677 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:18:59.943687 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:18:59.943733 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:18:59.943826 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.944344 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.945106 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.945308 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.945397 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.945552 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:18:59.945581 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:18:59.945637 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 08:18:59.945688 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 08:18:59.945706 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:18:59.945807 6984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:18:59.943439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:42513 (local address 127.4.151.130:49695)
0619 08:18:59.943547 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 08:18:59.943558 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:18:59.943608 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:18:59.943740 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:18:59.943740 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:18:59.943742 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:18:59.943777 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:59.943778 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.944451 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.944452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:59.945017 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:18:59.945019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:18:59.945427 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:18:59.945428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:18:59.945474 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:18:59.945479 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:18:59.945636 (+ 157us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:18:59.945638 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:18:59.945686 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 08:18:59.945729 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 08:18:59.947762 6868 raft_consensus.cc:1261] T c68cc198778e4592a2d59e2439e71d04 P 7bf65bdee5064e3a9073783f51458a67 [term 1 FOLLOWER]: Refusing update from remote peer fae665b09029444180774fb93b894a07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:18:59.947793 6588 raft_consensus.cc:1261] T c68cc198778e4592a2d59e2439e71d04 P 34bcc72af5fa41369e350065f960e11a [term 1 FOLLOWER]: Refusing update from remote peer fae665b09029444180774fb93b894a07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:18:59.948000 6943 consensus_queue.cc:1035] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:18:59.948112 6995 consensus_queue.cc:1035] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:18:59.949018 6969 mvcc.cc:205] Tried to move back new op lower bound from 7298486845224964096 to 7298486845100634112. Current Snapshot: MvccSnapshot[applied={T|T < 7298486845224964096}]
W20260619 08:18:59.953044 6969 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 08:19:01.471796 6431 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:19:01.471992 6431 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:19:01.472932 6431 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652: Sending DeleteTablet for 3 replicas of tablet c68cc198778e4592a2d59e2439e71d04
I20260619 08:19:01.473204 6707 tablet_service.cc:1514] Processing DeleteTablet for tablet c68cc198778e4592a2d59e2439e71d04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:01 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:55348
I20260619 08:19:01.473234 6568 tablet_service.cc:1514] Processing DeleteTablet for tablet c68cc198778e4592a2d59e2439e71d04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:01 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:37420
I20260619 08:19:01.473258 6848 tablet_service.cc:1514] Processing DeleteTablet for tablet c68cc198778e4592a2d59e2439e71d04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:01 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:58966
I20260619 08:19:01.473479 7020 tablet_replica.cc:330] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07: stopping tablet replica
I20260619 08:19:01.473562 7020 raft_consensus.cc:2229] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:19:01.473694 7020 raft_consensus.cc:2258] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:01.474061 7020 ts_tablet_manager.cc:1905] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:01.475296 7022 tablet_replica.cc:330] T c68cc198778e4592a2d59e2439e71d04 P 7bf65bdee5064e3a9073783f51458a67: stopping tablet replica
I20260619 08:19:01.475380 7022 raft_consensus.cc:2229] T c68cc198778e4592a2d59e2439e71d04 P 7bf65bdee5064e3a9073783f51458a67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:19:01.475426 7022 raft_consensus.cc:2258] T c68cc198778e4592a2d59e2439e71d04 P 7bf65bdee5064e3a9073783f51458a67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:01.475533 7021 tablet_replica.cc:330] T c68cc198778e4592a2d59e2439e71d04 P 34bcc72af5fa41369e350065f960e11a: stopping tablet replica
I20260619 08:19:01.475605 7021 raft_consensus.cc:2229] T c68cc198778e4592a2d59e2439e71d04 P 34bcc72af5fa41369e350065f960e11a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:19:01.475648 7021 raft_consensus.cc:2258] T c68cc198778e4592a2d59e2439e71d04 P 34bcc72af5fa41369e350065f960e11a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:01.475909 7021 ts_tablet_manager.cc:1905] T c68cc198778e4592a2d59e2439e71d04 P 34bcc72af5fa41369e350065f960e11a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:01.475922 7022 ts_tablet_manager.cc:1905] T c68cc198778e4592a2d59e2439e71d04 P 7bf65bdee5064e3a9073783f51458a67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:01.476199 7020 ts_tablet_manager.cc:1918] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 08:19:01.476310 7020 log.cc:1196] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/c68cc198778e4592a2d59e2439e71d04
I20260619 08:19:01.476590 7020 ts_tablet_manager.cc:1939] T c68cc198778e4592a2d59e2439e71d04 P fae665b09029444180774fb93b894a07: Deleting consensus metadata
I20260619 08:19:01.476935 6418 catalog_manager.cc:4841] TS fae665b09029444180774fb93b894a07 (127.4.151.130:42513): tablet c68cc198778e4592a2d59e2439e71d04 (table test-table [id=1ce2cb6929f14d2195868ec9d6f41af2]) successfully deleted
I20260619 08:19:01.477741 7022 ts_tablet_manager.cc:1918] T c68cc198778e4592a2d59e2439e71d04 P 7bf65bdee5064e3a9073783f51458a67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 08:19:01.477806 7022 log.cc:1196] T c68cc198778e4592a2d59e2439e71d04 P 7bf65bdee5064e3a9073783f51458a67: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/c68cc198778e4592a2d59e2439e71d04
I20260619 08:19:01.478024 7022 ts_tablet_manager.cc:1939] T c68cc198778e4592a2d59e2439e71d04 P 7bf65bdee5064e3a9073783f51458a67: Deleting consensus metadata
I20260619 08:19:01.478370 7021 ts_tablet_manager.cc:1918] T c68cc198778e4592a2d59e2439e71d04 P 34bcc72af5fa41369e350065f960e11a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 08:19:01.478488 7021 log.cc:1196] T c68cc198778e4592a2d59e2439e71d04 P 34bcc72af5fa41369e350065f960e11a: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-0/wal/wals/c68cc198778e4592a2d59e2439e71d04
I20260619 08:19:01.478608 6415 catalog_manager.cc:4841] TS 7bf65bdee5064e3a9073783f51458a67 (127.4.151.131:40785): tablet c68cc198778e4592a2d59e2439e71d04 (table test-table [id=1ce2cb6929f14d2195868ec9d6f41af2]) successfully deleted
I20260619 08:19:01.478758 7021 ts_tablet_manager.cc:1939] T c68cc198778e4592a2d59e2439e71d04 P 34bcc72af5fa41369e350065f960e11a: Deleting consensus metadata
I20260619 08:19:01.479130 6417 catalog_manager.cc:4841] TS 34bcc72af5fa41369e350065f960e11a (127.4.151.129:42561): tablet c68cc198778e4592a2d59e2439e71d04 (table test-table [id=1ce2cb6929f14d2195868ec9d6f41af2]) successfully deleted
Jun 19 08:19:01 dist-test-slave-d3n8 krb5kdc[6386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857141, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 08:19:01 dist-test-slave-d3n8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857141, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:01.504462 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:01.491491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58454 (local address 127.4.151.190:45777)
0619 08:19:01.491647 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:01.491649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:01.491661 (+ 12us) server_negotiation.cc:407] Connection header received
0619 08:19:01.491687 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:01.491688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:01.491697 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:01.491770 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:01.492338 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:01.492855 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:01.493753 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:01.493970 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:01.494037 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:01.501867 (+ 7830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:01.501869 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:01.501869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:01.501876 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:01.503426 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:01.503773 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:01.503774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:01.503774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:01.503828 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:01.504003 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:01.504004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:01.504004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:01.504152 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:01.504232 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:01.504322 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 08:19:01.504389 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":100,"threads_started":1}
I20260619 08:19:01.506204 6431 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:19:01.506402 6431 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:19:01.507983 6707 tablet_service.cc:1467] Processing CreateTablet for tablet c2aee975da0148839f8d80b4e1b4a128 (DEFAULT_TABLE table=test-table [id=b035f07989e542c2ad61a58956db3554]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:01.507983 6568 tablet_service.cc:1467] Processing CreateTablet for tablet c2aee975da0148839f8d80b4e1b4a128 (DEFAULT_TABLE table=test-table [id=b035f07989e542c2ad61a58956db3554]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:01.507988 6848 tablet_service.cc:1467] Processing CreateTablet for tablet c2aee975da0148839f8d80b4e1b4a128 (DEFAULT_TABLE table=test-table [id=b035f07989e542c2ad61a58956db3554]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:01.508127 6848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2aee975da0148839f8d80b4e1b4a128. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:01.508131 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2aee975da0148839f8d80b4e1b4a128. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:01.508126 6707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2aee975da0148839f8d80b4e1b4a128. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:01.509196 7031 tablet_bootstrap.cc:492] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67: Bootstrap starting.
I20260619 08:19:01.509649 7031 tablet_bootstrap.cc:654] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:01.509650 7032 tablet_bootstrap.cc:492] T c2aee975da0148839f8d80b4e1b4a128 P 34bcc72af5fa41369e350065f960e11a: Bootstrap starting.
I20260619 08:19:01.510059 7032 tablet_bootstrap.cc:654] T c2aee975da0148839f8d80b4e1b4a128 P 34bcc72af5fa41369e350065f960e11a: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:01.510227 7031 tablet_bootstrap.cc:492] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67: No bootstrap required, opened a new log
I20260619 08:19:01.510272 7031 ts_tablet_manager.cc:1397] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:01.510424 7033 tablet_bootstrap.cc:492] T c2aee975da0148839f8d80b4e1b4a128 P fae665b09029444180774fb93b894a07: Bootstrap starting.
I20260619 08:19:01.510383 7031 raft_consensus.cc:348] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:19:01.510478 7031 raft_consensus.cc:374] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:01.510540 7031 raft_consensus.cc:729] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf65bdee5064e3a9073783f51458a67, State: Initialized, Role: FOLLOWER
I20260619 08:19:01.510613 7031 consensus_queue.cc:260] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:19:01.510695 7032 tablet_bootstrap.cc:492] T c2aee975da0148839f8d80b4e1b4a128 P 34bcc72af5fa41369e350065f960e11a: No bootstrap required, opened a new log
I20260619 08:19:01.510748 7032 ts_tablet_manager.cc:1397] T c2aee975da0148839f8d80b4e1b4a128 P 34bcc72af5fa41369e350065f960e11a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:01.510776 7033 tablet_bootstrap.cc:654] T c2aee975da0148839f8d80b4e1b4a128 P fae665b09029444180774fb93b894a07: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:01.510802 7031 ts_tablet_manager.cc:1428] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:01.510937 7032 raft_consensus.cc:348] T c2aee975da0148839f8d80b4e1b4a128 P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:19:01.511015 7032 raft_consensus.cc:374] T c2aee975da0148839f8d80b4e1b4a128 P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:01.511072 7032 raft_consensus.cc:729] T c2aee975da0148839f8d80b4e1b4a128 P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34bcc72af5fa41369e350065f960e11a, State: Initialized, Role: FOLLOWER
I20260619 08:19:01.511126 7032 consensus_queue.cc:260] T c2aee975da0148839f8d80b4e1b4a128 P 34bcc72af5fa41369e350065f960e11a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:19:01.511346 7032 ts_tablet_manager.cc:1428] T c2aee975da0148839f8d80b4e1b4a128 P 34bcc72af5fa41369e350065f960e11a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 08:19:01.511349 7033 tablet_bootstrap.cc:492] T c2aee975da0148839f8d80b4e1b4a128 P fae665b09029444180774fb93b894a07: No bootstrap required, opened a new log
I20260619 08:19:01.511412 7033 ts_tablet_manager.cc:1397] T c2aee975da0148839f8d80b4e1b4a128 P fae665b09029444180774fb93b894a07: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:01.511549 7033 raft_consensus.cc:348] T c2aee975da0148839f8d80b4e1b4a128 P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:19:01.511626 7033 raft_consensus.cc:374] T c2aee975da0148839f8d80b4e1b4a128 P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:01.511655 7033 raft_consensus.cc:729] T c2aee975da0148839f8d80b4e1b4a128 P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fae665b09029444180774fb93b894a07, State: Initialized, Role: FOLLOWER
I20260619 08:19:01.511713 7033 consensus_queue.cc:260] T c2aee975da0148839f8d80b4e1b4a128 P fae665b09029444180774fb93b894a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:19:01.511842 7033 ts_tablet_manager.cc:1428] T c2aee975da0148839f8d80b4e1b4a128 P fae665b09029444180774fb93b894a07: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 08:19:01.522895 7036 raft_consensus.cc:482] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:19:01.523007 7036 raft_consensus.cc:504] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:19:01.523200 7036 leader_election.cc:288] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34bcc72af5fa41369e350065f960e11a (127.4.151.129:42561), fae665b09029444180774fb93b894a07 (127.4.151.130:42513)
I20260619 08:19:01.523392 6588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2aee975da0148839f8d80b4e1b4a128" candidate_uuid: "7bf65bdee5064e3a9073783f51458a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34bcc72af5fa41369e350065f960e11a" is_pre_election: true
I20260619 08:19:01.523392 6728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2aee975da0148839f8d80b4e1b4a128" candidate_uuid: "7bf65bdee5064e3a9073783f51458a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fae665b09029444180774fb93b894a07" is_pre_election: true
I20260619 08:19:01.523504 6728 raft_consensus.cc:2454] T c2aee975da0148839f8d80b4e1b4a128 P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bf65bdee5064e3a9073783f51458a67 in term 0.
I20260619 08:19:01.523511 6588 raft_consensus.cc:2454] T c2aee975da0148839f8d80b4e1b4a128 P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bf65bdee5064e3a9073783f51458a67 in term 0.
I20260619 08:19:01.523694 6802 leader_election.cc:302] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bf65bdee5064e3a9073783f51458a67, fae665b09029444180774fb93b894a07; no voters:
I20260619 08:19:01.523783 7036 raft_consensus.cc:2781] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:19:01.523821 7036 raft_consensus.cc:482] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:19:01.523849 7036 raft_consensus.cc:3037] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:01.524549 7036 raft_consensus.cc:504] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:19:01.524681 7036 leader_election.cc:288] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67 [CANDIDATE]: Term 1 election: Requested vote from peers 34bcc72af5fa41369e350065f960e11a (127.4.151.129:42561), fae665b09029444180774fb93b894a07 (127.4.151.130:42513)
I20260619 08:19:01.524869 6588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2aee975da0148839f8d80b4e1b4a128" candidate_uuid: "7bf65bdee5064e3a9073783f51458a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34bcc72af5fa41369e350065f960e11a"
I20260619 08:19:01.524961 6588 raft_consensus.cc:3037] T c2aee975da0148839f8d80b4e1b4a128 P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:01.524834 6728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2aee975da0148839f8d80b4e1b4a128" candidate_uuid: "7bf65bdee5064e3a9073783f51458a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fae665b09029444180774fb93b894a07"
I20260619 08:19:01.525025 6728 raft_consensus.cc:3037] T c2aee975da0148839f8d80b4e1b4a128 P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:01.525703 6588 raft_consensus.cc:2454] T c2aee975da0148839f8d80b4e1b4a128 P 34bcc72af5fa41369e350065f960e11a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf65bdee5064e3a9073783f51458a67 in term 1.
I20260619 08:19:01.525792 6728 raft_consensus.cc:2454] T c2aee975da0148839f8d80b4e1b4a128 P fae665b09029444180774fb93b894a07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf65bdee5064e3a9073783f51458a67 in term 1.
I20260619 08:19:01.525879 6801 leader_election.cc:302] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34bcc72af5fa41369e350065f960e11a, 7bf65bdee5064e3a9073783f51458a67; no voters:
I20260619 08:19:01.525957 7036 raft_consensus.cc:2781] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:01.526116 7036 raft_consensus.cc:686] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67 [term 1 LEADER]: Becoming Leader. State: Replica: 7bf65bdee5064e3a9073783f51458a67, State: Running, Role: LEADER
I20260619 08:19:01.526165 7036 consensus_queue.cc:237] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:19:01.526726 6429 catalog_manager.cc:5495] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bf65bdee5064e3a9073783f51458a67 (127.4.151.131). New cstate: current_term: 1 leader_uuid: "7bf65bdee5064e3a9073783f51458a67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } health_report { overall_health: HEALTHY } } }
I20260619 08:19:01.537185 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:01.534529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35078 (local address 127.4.151.131:40785)
0619 08:19:01.534729 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:01.534731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:01.534739 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:19:01.534762 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:01.534762 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:01.534776 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:01.534837 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:19:01.535044 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:01.535720 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:01.536499 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:01.536696 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:01.536752 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:01.536915 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:19:01.536952 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:19:01.537010 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:01.537063 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 08:19:01.537085 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":123,"threads_started":1}
I20260619 08:19:01.539682 6728 raft_consensus.cc:1261] T c2aee975da0148839f8d80b4e1b4a128 P fae665b09029444180774fb93b894a07 [term 1 FOLLOWER]: Refusing update from remote peer 7bf65bdee5064e3a9073783f51458a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:19:01.539712 6588 raft_consensus.cc:1261] T c2aee975da0148839f8d80b4e1b4a128 P 34bcc72af5fa41369e350065f960e11a [term 1 FOLLOWER]: Refusing update from remote peer 7bf65bdee5064e3a9073783f51458a67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:19:01.539889 7039 consensus_queue.cc:1035] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:19:01.539983 7036 consensus_queue.cc:1035] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:19:01.549626 6431 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:19:01.549724 6431 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:19:01.550465 6431 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652: Sending DeleteTablet for 3 replicas of tablet c2aee975da0148839f8d80b4e1b4a128
I20260619 08:19:01.550716 6848 tablet_service.cc:1514] Processing DeleteTablet for tablet c2aee975da0148839f8d80b4e1b4a128 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:01 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:58966
I20260619 08:19:01.550741 6568 tablet_service.cc:1514] Processing DeleteTablet for tablet c2aee975da0148839f8d80b4e1b4a128 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:01 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:37420
I20260619 08:19:01.550837 7021 tablet_replica.cc:330] T c2aee975da0148839f8d80b4e1b4a128 P 34bcc72af5fa41369e350065f960e11a: stopping tablet replica
I20260619 08:19:01.550843 7022 tablet_replica.cc:330] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67: stopping tablet replica
I20260619 08:19:01.550832 6707 tablet_service.cc:1514] Processing DeleteTablet for tablet c2aee975da0148839f8d80b4e1b4a128 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:01 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:55348
I20260619 08:19:01.550899 7021 raft_consensus.cc:2229] T c2aee975da0148839f8d80b4e1b4a128 P 34bcc72af5fa41369e350065f960e11a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:19:01.550902 7022 raft_consensus.cc:2229] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:19:01.550912 7020 tablet_replica.cc:330] T c2aee975da0148839f8d80b4e1b4a128 P fae665b09029444180774fb93b894a07: stopping tablet replica
I20260619 08:19:01.550952 7021 raft_consensus.cc:2258] T c2aee975da0148839f8d80b4e1b4a128 P 34bcc72af5fa41369e350065f960e11a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:01.550966 7022 raft_consensus.cc:2258] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:01.550969 7020 raft_consensus.cc:2229] T c2aee975da0148839f8d80b4e1b4a128 P fae665b09029444180774fb93b894a07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:19:01.551003 7020 raft_consensus.cc:2258] T c2aee975da0148839f8d80b4e1b4a128 P fae665b09029444180774fb93b894a07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:01.551295 6431 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 08:19:01.551375 7020 ts_tablet_manager.cc:1905] T c2aee975da0148839f8d80b4e1b4a128 P fae665b09029444180774fb93b894a07: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 08:19:01.551455 6431 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:19:01.551518 7022 ts_tablet_manager.cc:1905] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:01.552634 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 3b68fcf0ae02449c991fe3eb65d98600 (DEFAULT_TABLE table=test-table [id=80874c0c50c543bcb48ec2a39fb7d118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:01.552744 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b68fcf0ae02449c991fe3eb65d98600. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:01.553231 7020 ts_tablet_manager.cc:1918] T c2aee975da0148839f8d80b4e1b4a128 P fae665b09029444180774fb93b894a07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:19:01.553287 7020 log.cc:1196] T c2aee975da0148839f8d80b4e1b4a128 P fae665b09029444180774fb93b894a07: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/c2aee975da0148839f8d80b4e1b4a128
I20260619 08:19:01.553561 7020 ts_tablet_manager.cc:1939] T c2aee975da0148839f8d80b4e1b4a128 P fae665b09029444180774fb93b894a07: Deleting consensus metadata
I20260619 08:19:01.553856 7022 ts_tablet_manager.cc:1918] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:19:01.553982 7022 log.cc:1196] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/c2aee975da0148839f8d80b4e1b4a128
I20260619 08:19:01.554030 7021 ts_tablet_manager.cc:1905] T c2aee975da0148839f8d80b4e1b4a128 P 34bcc72af5fa41369e350065f960e11a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:01.554315 6848 tablet_service.cc:1467] Processing CreateTablet for tablet 3b68fcf0ae02449c991fe3eb65d98600 (DEFAULT_TABLE table=test-table [id=80874c0c50c543bcb48ec2a39fb7d118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:01.554352 7022 ts_tablet_manager.cc:1939] T c2aee975da0148839f8d80b4e1b4a128 P 7bf65bdee5064e3a9073783f51458a67: Deleting consensus metadata
I20260619 08:19:01.554407 6848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b68fcf0ae02449c991fe3eb65d98600. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:01.554769 6707 tablet_service.cc:1467] Processing CreateTablet for tablet 3b68fcf0ae02449c991fe3eb65d98600 (DEFAULT_TABLE table=test-table [id=80874c0c50c543bcb48ec2a39fb7d118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:01.554857 6707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b68fcf0ae02449c991fe3eb65d98600. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:01.555683 7021 ts_tablet_manager.cc:1918] T c2aee975da0148839f8d80b4e1b4a128 P 34bcc72af5fa41369e350065f960e11a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:19:01.555745 7021 log.cc:1196] T c2aee975da0148839f8d80b4e1b4a128 P 34bcc72af5fa41369e350065f960e11a: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-0/wal/wals/c2aee975da0148839f8d80b4e1b4a128
I20260619 08:19:01.555882 7032 tablet_bootstrap.cc:492] T 3b68fcf0ae02449c991fe3eb65d98600 P 34bcc72af5fa41369e350065f960e11a: Bootstrap starting.
I20260619 08:19:01.555949 7021 ts_tablet_manager.cc:1939] T c2aee975da0148839f8d80b4e1b4a128 P 34bcc72af5fa41369e350065f960e11a: Deleting consensus metadata
I20260619 08:19:01.556278 7031 tablet_bootstrap.cc:492] T 3b68fcf0ae02449c991fe3eb65d98600 P 7bf65bdee5064e3a9073783f51458a67: Bootstrap starting.
I20260619 08:19:01.556432 6415 catalog_manager.cc:4841] TS 7bf65bdee5064e3a9073783f51458a67 (127.4.151.131:40785): tablet c2aee975da0148839f8d80b4e1b4a128 (table test-table [id=b035f07989e542c2ad61a58956db3554]) successfully deleted
I20260619 08:19:01.556604 7031 tablet_bootstrap.cc:654] T 3b68fcf0ae02449c991fe3eb65d98600 P 7bf65bdee5064e3a9073783f51458a67: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:01.556620 7032 tablet_bootstrap.cc:654] T 3b68fcf0ae02449c991fe3eb65d98600 P 34bcc72af5fa41369e350065f960e11a: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:01.556928 6418 catalog_manager.cc:4841] TS fae665b09029444180774fb93b894a07 (127.4.151.130:42513): tablet c2aee975da0148839f8d80b4e1b4a128 (table test-table [id=b035f07989e542c2ad61a58956db3554]) successfully deleted
I20260619 08:19:01.557214 6417 catalog_manager.cc:4841] TS 34bcc72af5fa41369e350065f960e11a (127.4.151.129:42561): tablet c2aee975da0148839f8d80b4e1b4a128 (table test-table [id=b035f07989e542c2ad61a58956db3554]) successfully deleted
I20260619 08:19:01.557305 7031 tablet_bootstrap.cc:492] T 3b68fcf0ae02449c991fe3eb65d98600 P 7bf65bdee5064e3a9073783f51458a67: No bootstrap required, opened a new log
I20260619 08:19:01.557395 7031 ts_tablet_manager.cc:1397] T 3b68fcf0ae02449c991fe3eb65d98600 P 7bf65bdee5064e3a9073783f51458a67: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:01.557556 7031 raft_consensus.cc:348] T 3b68fcf0ae02449c991fe3eb65d98600 P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:19:01.557643 7031 raft_consensus.cc:374] T 3b68fcf0ae02449c991fe3eb65d98600 P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:01.557675 7031 raft_consensus.cc:729] T 3b68fcf0ae02449c991fe3eb65d98600 P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf65bdee5064e3a9073783f51458a67, State: Initialized, Role: FOLLOWER
I20260619 08:19:01.557719 7031 consensus_queue.cc:260] T 3b68fcf0ae02449c991fe3eb65d98600 P 7bf65bdee5064e3a9073783f51458a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:19:01.557812 7031 ts_tablet_manager.cc:1428] T 3b68fcf0ae02449c991fe3eb65d98600 P 7bf65bdee5064e3a9073783f51458a67: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:01.557988 7033 tablet_bootstrap.cc:492] T 3b68fcf0ae02449c991fe3eb65d98600 P fae665b09029444180774fb93b894a07: Bootstrap starting.
I20260619 08:19:01.558290 7033 tablet_bootstrap.cc:654] T 3b68fcf0ae02449c991fe3eb65d98600 P fae665b09029444180774fb93b894a07: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:01.558298 7032 tablet_bootstrap.cc:492] T 3b68fcf0ae02449c991fe3eb65d98600 P 34bcc72af5fa41369e350065f960e11a: No bootstrap required, opened a new log
I20260619 08:19:01.558348 7032 ts_tablet_manager.cc:1397] T 3b68fcf0ae02449c991fe3eb65d98600 P 34bcc72af5fa41369e350065f960e11a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:19:01.558493 7032 raft_consensus.cc:348] T 3b68fcf0ae02449c991fe3eb65d98600 P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:19:01.558571 7032 raft_consensus.cc:374] T 3b68fcf0ae02449c991fe3eb65d98600 P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:01.558599 7032 raft_consensus.cc:729] T 3b68fcf0ae02449c991fe3eb65d98600 P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34bcc72af5fa41369e350065f960e11a, State: Initialized, Role: FOLLOWER
I20260619 08:19:01.558643 7032 consensus_queue.cc:260] T 3b68fcf0ae02449c991fe3eb65d98600 P 34bcc72af5fa41369e350065f960e11a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:19:01.558737 7032 ts_tablet_manager.cc:1428] T 3b68fcf0ae02449c991fe3eb65d98600 P 34bcc72af5fa41369e350065f960e11a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:01.559099 7033 tablet_bootstrap.cc:492] T 3b68fcf0ae02449c991fe3eb65d98600 P fae665b09029444180774fb93b894a07: No bootstrap required, opened a new log
I20260619 08:19:01.559150 7033 ts_tablet_manager.cc:1397] T 3b68fcf0ae02449c991fe3eb65d98600 P fae665b09029444180774fb93b894a07: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:01.559268 7033 raft_consensus.cc:348] T 3b68fcf0ae02449c991fe3eb65d98600 P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:19:01.559341 7033 raft_consensus.cc:374] T 3b68fcf0ae02449c991fe3eb65d98600 P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:01.559361 7033 raft_consensus.cc:729] T 3b68fcf0ae02449c991fe3eb65d98600 P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fae665b09029444180774fb93b894a07, State: Initialized, Role: FOLLOWER
I20260619 08:19:01.559417 7033 consensus_queue.cc:260] T 3b68fcf0ae02449c991fe3eb65d98600 P fae665b09029444180774fb93b894a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:19:01.559511 7033 ts_tablet_manager.cc:1428] T 3b68fcf0ae02449c991fe3eb65d98600 P fae665b09029444180774fb93b894a07: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:01.605983 7038 raft_consensus.cc:482] T 3b68fcf0ae02449c991fe3eb65d98600 P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:19:01.606115 7038 raft_consensus.cc:504] T 3b68fcf0ae02449c991fe3eb65d98600 P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:19:01.606462 7038 leader_election.cc:288] T 3b68fcf0ae02449c991fe3eb65d98600 P 34bcc72af5fa41369e350065f960e11a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fae665b09029444180774fb93b894a07 (127.4.151.130:42513), 7bf65bdee5064e3a9073783f51458a67 (127.4.151.131:40785)
I20260619 08:19:01.610072 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:01.606732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:34277 (local address 127.4.151.130:42513)
0619 08:19:01.606836 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:01.606838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:01.606848 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:19:01.606977 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:01.606978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:01.606989 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:01.607067 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:01.607261 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:01.607758 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:01.609307 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:01.609711 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:01.609795 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:01.609806 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:01.609955 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 08:19:01.609996 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":58,"threads_started":1}
I20260619 08:19:01.610106 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:01.606701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:60461 (local address 127.4.151.131:40785)
0619 08:19:01.606748 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:01.606750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:01.606919 (+ 169us) server_negotiation.cc:407] Connection header received
0619 08:19:01.607018 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:01.607020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:01.607035 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:01.607104 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:01.607281 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:01.607757 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:01.609291 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:01.609728 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:01.609811 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:01.609820 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:01.610002 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 08:19:01.610037 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:19:01.610126 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:01.606633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:40785 (local address 127.4.151.129:60461)
0619 08:19:01.606854 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:01.606866 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:01.606937 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:01.607120 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:01.607121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:01.607124 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:01.607201 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:01.607203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:01.607842 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:01.607843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:01.609193 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:01.609196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:01.609810 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:01.609812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:01.609918 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:01.609922 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:19:01.609994 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 08:19:01.610037 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":61,"thread_start_us":63,"threads_started":1}
I20260619 08:19:01.610105 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:01.606633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:42513 (local address 127.4.151.129:34277)
0619 08:19:01.606781 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:01.606793 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:01.606845 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:01.607082 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:01.607083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:01.607086 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:01.607189 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:01.607191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:01.607909 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:01.607910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:01.609193 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:01.609196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:01.609801 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:01.609802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:01.609881 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:01.609885 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:19:01.609948 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 08:19:01.609992 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":56,"thread_start_us":90,"threads_started":1}
I20260619 08:19:01.610545 6867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b68fcf0ae02449c991fe3eb65d98600" candidate_uuid: "34bcc72af5fa41369e350065f960e11a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf65bdee5064e3a9073783f51458a67" is_pre_election: true
I20260619 08:19:01.610634 6867 raft_consensus.cc:2454] T 3b68fcf0ae02449c991fe3eb65d98600 P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34bcc72af5fa41369e350065f960e11a in term 0.
I20260619 08:19:01.610822 6519 leader_election.cc:302] T 3b68fcf0ae02449c991fe3eb65d98600 P 34bcc72af5fa41369e350065f960e11a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34bcc72af5fa41369e350065f960e11a, 7bf65bdee5064e3a9073783f51458a67; no voters:
I20260619 08:19:01.610808 6728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b68fcf0ae02449c991fe3eb65d98600" candidate_uuid: "34bcc72af5fa41369e350065f960e11a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fae665b09029444180774fb93b894a07" is_pre_election: true
I20260619 08:19:01.610955 6728 raft_consensus.cc:2454] T 3b68fcf0ae02449c991fe3eb65d98600 P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34bcc72af5fa41369e350065f960e11a in term 0.
I20260619 08:19:01.610975 7038 raft_consensus.cc:2781] T 3b68fcf0ae02449c991fe3eb65d98600 P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:19:01.611016 7038 raft_consensus.cc:482] T 3b68fcf0ae02449c991fe3eb65d98600 P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:19:01.611059 7038 raft_consensus.cc:3037] T 3b68fcf0ae02449c991fe3eb65d98600 P 34bcc72af5fa41369e350065f960e11a [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:01.611773 7038 raft_consensus.cc:504] T 3b68fcf0ae02449c991fe3eb65d98600 P 34bcc72af5fa41369e350065f960e11a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:19:01.611907 7038 leader_election.cc:288] T 3b68fcf0ae02449c991fe3eb65d98600 P 34bcc72af5fa41369e350065f960e11a [CANDIDATE]: Term 1 election: Requested vote from peers fae665b09029444180774fb93b894a07 (127.4.151.130:42513), 7bf65bdee5064e3a9073783f51458a67 (127.4.151.131:40785)
I20260619 08:19:01.612053 6867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b68fcf0ae02449c991fe3eb65d98600" candidate_uuid: "34bcc72af5fa41369e350065f960e11a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf65bdee5064e3a9073783f51458a67"
I20260619 08:19:01.612063 6728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b68fcf0ae02449c991fe3eb65d98600" candidate_uuid: "34bcc72af5fa41369e350065f960e11a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fae665b09029444180774fb93b894a07"
I20260619 08:19:01.612131 6728 raft_consensus.cc:3037] T 3b68fcf0ae02449c991fe3eb65d98600 P fae665b09029444180774fb93b894a07 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:01.612131 6867 raft_consensus.cc:3037] T 3b68fcf0ae02449c991fe3eb65d98600 P 7bf65bdee5064e3a9073783f51458a67 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:01.612876 6867 raft_consensus.cc:2454] T 3b68fcf0ae02449c991fe3eb65d98600 P 7bf65bdee5064e3a9073783f51458a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34bcc72af5fa41369e350065f960e11a in term 1.
I20260619 08:19:01.612876 6728 raft_consensus.cc:2454] T 3b68fcf0ae02449c991fe3eb65d98600 P fae665b09029444180774fb93b894a07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34bcc72af5fa41369e350065f960e11a in term 1.
I20260619 08:19:01.613027 6519 leader_election.cc:302] T 3b68fcf0ae02449c991fe3eb65d98600 P 34bcc72af5fa41369e350065f960e11a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34bcc72af5fa41369e350065f960e11a, 7bf65bdee5064e3a9073783f51458a67; no voters:
I20260619 08:19:01.613130 7038 raft_consensus.cc:2781] T 3b68fcf0ae02449c991fe3eb65d98600 P 34bcc72af5fa41369e350065f960e11a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:01.613263 7038 raft_consensus.cc:686] T 3b68fcf0ae02449c991fe3eb65d98600 P 34bcc72af5fa41369e350065f960e11a [term 1 LEADER]: Becoming Leader. State: Replica: 34bcc72af5fa41369e350065f960e11a, State: Running, Role: LEADER
I20260619 08:19:01.613337 7038 consensus_queue.cc:237] T 3b68fcf0ae02449c991fe3eb65d98600 P 34bcc72af5fa41369e350065f960e11a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } }
I20260619 08:19:01.614084 6429 catalog_manager.cc:5495] T 3b68fcf0ae02449c991fe3eb65d98600 P 34bcc72af5fa41369e350065f960e11a reported cstate change: term changed from 0 to 1, leader changed from <none> to 34bcc72af5fa41369e350065f960e11a (127.4.151.129). New cstate: current_term: 1 leader_uuid: "34bcc72af5fa41369e350065f960e11a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34bcc72af5fa41369e350065f960e11a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 } health_report { overall_health: UNKNOWN } } }
I20260619 08:19:01.624522 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:01.621931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57618 (local address 127.4.151.129:42561)
0619 08:19:01.622099 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:01.622101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:01.622107 (+ 6us) server_negotiation.cc:407] Connection header received
0619 08:19:01.622135 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:01.622135 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:01.622152 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:01.622233 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:19:01.622342 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:01.622843 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:01.623682 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:01.623894 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:01.623981 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:01.624143 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:19:01.624260 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:19:01.624343 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:01.624382 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 08:19:01.624402 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":98,"threads_started":1}
I20260619 08:19:01.629539 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:01.627251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:40977 (local address 127.4.151.130:42513)
0619 08:19:01.627327 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:01.627329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:01.627339 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:19:01.627403 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:01.627404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:01.627413 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:01.627468 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:19:01.627573 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:01.628011 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:01.628799 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:01.628980 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:01.629070 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:01.629268 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:19:01.629293 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:19:01.629332 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:01.629454 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 08:19:01.629479 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 08:19:01.629566 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:01.627087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:42513 (local address 127.4.151.129:40977)
0619 08:19:01.627275 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:01.627288 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:01.627358 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:01.627477 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:01.627478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:01.627480 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:19:01.627532 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:01.627533 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:01.628140 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:01.628141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:01.628732 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:01.628733 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:01.629102 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:01.629103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:01.629176 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:01.629190 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:19:01.629344 (+ 154us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:19:01.629346 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:19:01.629445 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 08:19:01.629490 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":72,"threads_started":1}
I20260619 08:19:01.631588 6867 raft_consensus.cc:1261] T 3b68fcf0ae02449c991fe3eb65d98600 P 7bf65bdee5064e3a9073783f51458a67 [term 1 FOLLOWER]: Refusing update from remote peer 34bcc72af5fa41369e350065f960e11a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:19:01.631588 6728 raft_consensus.cc:1261] T 3b68fcf0ae02449c991fe3eb65d98600 P fae665b09029444180774fb93b894a07 [term 1 FOLLOWER]: Refusing update from remote peer 34bcc72af5fa41369e350065f960e11a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:19:01.631843 7055 consensus_queue.cc:1035] T 3b68fcf0ae02449c991fe3eb65d98600 P 34bcc72af5fa41369e350065f960e11a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fae665b09029444180774fb93b894a07" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:19:01.632010 7038 consensus_queue.cc:1035] T 3b68fcf0ae02449c991fe3eb65d98600 P 34bcc72af5fa41369e350065f960e11a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf65bdee5064e3a9073783f51458a67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:19:01.639109 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:01.636680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:41445 (local address 127.4.151.129:42561)
0619 08:19:01.636742 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:01.636744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:01.636853 (+ 109us) server_negotiation.cc:407] Connection header received
0619 08:19:01.636923 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:01.636923 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:01.636935 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:01.636991 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:19:01.637089 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:01.637637 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:01.638486 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:01.638721 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:01.638782 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:01.638919 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:19:01.638952 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:19:01.638993 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:01.639039 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 08:19:01.639058 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 08:19:01.639132 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:01.636626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:42561 (local address 127.4.151.130:41445)
0619 08:19:01.636783 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:01.636795 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:01.636854 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:01.636994 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:01.636995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:01.636997 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:19:01.637049 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:01.637050 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:01.637724 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:01.637725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:01.638387 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:01.638393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:01.638780 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:01.638781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:01.638844 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:01.638850 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:19:01.638993 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:19:01.638993 (+ 0us) client_negotiation.cc:769] Sending connection context
0619 08:19:01.639034 (+ 41us) client_negotiation.cc:240] Negotiation successful
0619 08:19:01.639072 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":106,"threads_started":1}
W20260619 08:19:01.639460 7006 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 08:19:03.163380 6429 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:19:03.163528 6429 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:19:03.164382 6429 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a725ef00170490ab4763e0cc7297652: Sending DeleteTablet for 3 replicas of tablet 3b68fcf0ae02449c991fe3eb65d98600
I20260619 08:19:03.164674 6707 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b68fcf0ae02449c991fe3eb65d98600 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:03 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:55348
I20260619 08:19:03.164681 6848 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b68fcf0ae02449c991fe3eb65d98600 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:03 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:58966
I20260619 08:19:03.164840 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 6503
I20260619 08:19:03.167884 7088 tablet_replica.cc:330] T 3b68fcf0ae02449c991fe3eb65d98600 P 7bf65bdee5064e3a9073783f51458a67: stopping tablet replica
I20260619 08:19:03.167972 7088 raft_consensus.cc:2229] T 3b68fcf0ae02449c991fe3eb65d98600 P 7bf65bdee5064e3a9073783f51458a67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:19:03.168015 7088 raft_consensus.cc:2258] T 3b68fcf0ae02449c991fe3eb65d98600 P 7bf65bdee5064e3a9073783f51458a67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:03.168121 7089 tablet_replica.cc:330] T 3b68fcf0ae02449c991fe3eb65d98600 P fae665b09029444180774fb93b894a07: stopping tablet replica
I20260619 08:19:03.168185 7089 raft_consensus.cc:2229] T 3b68fcf0ae02449c991fe3eb65d98600 P fae665b09029444180774fb93b894a07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:19:03.168227 7089 raft_consensus.cc:2258] T 3b68fcf0ae02449c991fe3eb65d98600 P fae665b09029444180774fb93b894a07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:03.168450 7088 ts_tablet_manager.cc:1905] T 3b68fcf0ae02449c991fe3eb65d98600 P 7bf65bdee5064e3a9073783f51458a67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:03.168470 7089 ts_tablet_manager.cc:1905] T 3b68fcf0ae02449c991fe3eb65d98600 P fae665b09029444180774fb93b894a07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:03.170154 7089 ts_tablet_manager.cc:1918] T 3b68fcf0ae02449c991fe3eb65d98600 P fae665b09029444180774fb93b894a07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 08:19:03.170225 7089 log.cc:1196] T 3b68fcf0ae02449c991fe3eb65d98600 P fae665b09029444180774fb93b894a07: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/3b68fcf0ae02449c991fe3eb65d98600
I20260619 08:19:03.170471 7088 ts_tablet_manager.cc:1918] T 3b68fcf0ae02449c991fe3eb65d98600 P 7bf65bdee5064e3a9073783f51458a67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 08:19:03.170490 7089 ts_tablet_manager.cc:1939] T 3b68fcf0ae02449c991fe3eb65d98600 P fae665b09029444180774fb93b894a07: Deleting consensus metadata
I20260619 08:19:03.170537 7088 log.cc:1196] T 3b68fcf0ae02449c991fe3eb65d98600 P 7bf65bdee5064e3a9073783f51458a67: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/3b68fcf0ae02449c991fe3eb65d98600
I20260619 08:19:03.170749 7088 ts_tablet_manager.cc:1939] T 3b68fcf0ae02449c991fe3eb65d98600 P 7bf65bdee5064e3a9073783f51458a67: Deleting consensus metadata
I20260619 08:19:03.170883 6418 catalog_manager.cc:4841] TS fae665b09029444180774fb93b894a07 (127.4.151.130:42513): tablet 3b68fcf0ae02449c991fe3eb65d98600 (table test-table [id=80874c0c50c543bcb48ec2a39fb7d118]) successfully deleted
I20260619 08:19:03.171613 6415 catalog_manager.cc:4841] TS 7bf65bdee5064e3a9073783f51458a67 (127.4.151.131:40785): tablet 3b68fcf0ae02449c991fe3eb65d98600 (table test-table [id=80874c0c50c543bcb48ec2a39fb7d118]) successfully deleted
W20260619 08:19:03.172468 6417 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.151.129:42561 (error 108)
I20260619 08:19:03.172729 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 6643
I20260619 08:19:03.174067 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:03.172830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:42561 (local address 127.0.0.1:57634)
0619 08:19:03.173985 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:03.174030 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.129:42561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1122,"thread_start_us":67,"threads_started":1}
W20260619 08:19:03.174181 6417 catalog_manager.cc:4584] TS 34bcc72af5fa41369e350065f960e11a (127.4.151.129:42561): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3b68fcf0ae02449c991fe3eb65d98600: Network error: Client connection negotiation failed: client connection to 127.4.151.129:42561: connect: Connection refused (error 111)
I20260619 08:19:03.178725 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 6783
I20260619 08:19:03.184221 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 6399
2026-06-19T08:19:03Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6051 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:19:03 dist-test-slave-d3n8 krb5kdc[7094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:19:03 dist-test-slave-d3n8 krb5kdc[7094](info): set up 2 sockets
Jun 19 08:19:03 dist-test-slave-d3n8 krb5kdc[7094](info): commencing operation
krb5kdc: starting...
W20260619 08:19:04.237593 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:19:04 dist-test-slave-d3n8 krb5kdc[7094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:19:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:19:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:19:04.364588 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:43157
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:43157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:04.451062 7107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:04.451333 7107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:04.451375 7107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:04.452576 7107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:19:04.452657 7107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:04.452682 7107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:04.452706 7107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:19:04.452734 7107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:19:04.454273 7107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:43157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:43157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:04.454571 7107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:04.454813 7107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:04.459872 7115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:04.459883 7113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:04.460021 7107 server_base.cc:1034] running on GCE node
W20260619 08:19:04.460022 7112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:04.460567 7107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:04.460815 7107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:04.462028 7107 hybrid_clock.cc:648] HybridClock initialized: now 1781857144461958 us; error 78 us; skew 500 ppm
Jun 19 08:19:04 dist-test-slave-d3n8 krb5kdc[7094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857144, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:04.464569 7107 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:19:04.465023 7107 webserver.cc:460] Webserver started at http://127.4.151.190:42831/ using document root <none> and password file <none>
I20260619 08:19:04.465276 7107 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:04.465345 7107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:04.465495 7107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:04.466678 7107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "df81c885ec934e67b7288278ccc5c680"
format_stamp: "Formatted at 2026-06-19 08:19:04 on dist-test-slave-d3n8"
server_key: "de0833377fa3d47c2298100060ea1322"
server_key_iv: "085873944e8a445f21168db94f83f738"
server_key_version: "encryptionkey@0"
I20260619 08:19:04.467065 7107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "df81c885ec934e67b7288278ccc5c680"
format_stamp: "Formatted at 2026-06-19 08:19:04 on dist-test-slave-d3n8"
server_key: "de0833377fa3d47c2298100060ea1322"
server_key_iv: "085873944e8a445f21168db94f83f738"
server_key_version: "encryptionkey@0"
I20260619 08:19:04.468461 7107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:19:04.469326 7122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:04.469583 7107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:04.469696 7107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "df81c885ec934e67b7288278ccc5c680"
format_stamp: "Formatted at 2026-06-19 08:19:04 on dist-test-slave-d3n8"
server_key: "de0833377fa3d47c2298100060ea1322"
server_key_iv: "085873944e8a445f21168db94f83f738"
server_key_version: "encryptionkey@0"
I20260619 08:19:04.469806 7107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:04.478632 7107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:04.480985 7107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:04.481159 7107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:04.485088 7107 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:43157
I20260619 08:19:04.485124 7174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:43157 every 8 connection(s)
I20260619 08:19:04.485497 7107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:19:04.486414 7175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:04.488605 7175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df81c885ec934e67b7288278ccc5c680: Bootstrap starting.
I20260619 08:19:04.489132 7175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df81c885ec934e67b7288278ccc5c680: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:04.489362 7175 log.cc:824] T 00000000000000000000000000000000 P df81c885ec934e67b7288278ccc5c680: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:04.490134 7175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df81c885ec934e67b7288278ccc5c680: No bootstrap required, opened a new log
I20260619 08:19:04.490324 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 7107
I20260619 08:19:04.490461 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:19:04.490633 4702 external_mini_cluster.cc:1428] Setting key f422191d5589fe5608b23a2a4ac03908
I20260619 08:19:04.492218 7175 raft_consensus.cc:348] T 00000000000000000000000000000000 P df81c885ec934e67b7288278ccc5c680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df81c885ec934e67b7288278ccc5c680" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43157 } }
I20260619 08:19:04.492386 7175 raft_consensus.cc:374] T 00000000000000000000000000000000 P df81c885ec934e67b7288278ccc5c680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:04.492451 7175 raft_consensus.cc:729] T 00000000000000000000000000000000 P df81c885ec934e67b7288278ccc5c680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df81c885ec934e67b7288278ccc5c680, State: Initialized, Role: FOLLOWER
I20260619 08:19:04.492581 7175 consensus_queue.cc:260] T 00000000000000000000000000000000 P df81c885ec934e67b7288278ccc5c680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df81c885ec934e67b7288278ccc5c680" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43157 } }
I20260619 08:19:04.492686 7175 raft_consensus.cc:388] T 00000000000000000000000000000000 P df81c885ec934e67b7288278ccc5c680 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:19:04.492739 7175 raft_consensus.cc:482] T 00000000000000000000000000000000 P df81c885ec934e67b7288278ccc5c680 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:19:04.492791 7175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df81c885ec934e67b7288278ccc5c680 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:04.493687 7175 raft_consensus.cc:504] T 00000000000000000000000000000000 P df81c885ec934e67b7288278ccc5c680 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df81c885ec934e67b7288278ccc5c680" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43157 } }
I20260619 08:19:04.493809 7175 leader_election.cc:302] T 00000000000000000000000000000000 P df81c885ec934e67b7288278ccc5c680 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df81c885ec934e67b7288278ccc5c680; no voters:
I20260619 08:19:04.494022 7175 leader_election.cc:288] T 00000000000000000000000000000000 P df81c885ec934e67b7288278ccc5c680 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:19:04.494057 7180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df81c885ec934e67b7288278ccc5c680 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:04.494164 7180 raft_consensus.cc:686] T 00000000000000000000000000000000 P df81c885ec934e67b7288278ccc5c680 [term 1 LEADER]: Becoming Leader. State: Replica: df81c885ec934e67b7288278ccc5c680, State: Running, Role: LEADER
I20260619 08:19:04.494289 7175 sys_catalog.cc:564] T 00000000000000000000000000000000 P df81c885ec934e67b7288278ccc5c680 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:19:04.494310 7180 consensus_queue.cc:237] T 00000000000000000000000000000000 P df81c885ec934e67b7288278ccc5c680 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df81c885ec934e67b7288278ccc5c680" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43157 } }
I20260619 08:19:04.494724 7180 sys_catalog.cc:455] T 00000000000000000000000000000000 P df81c885ec934e67b7288278ccc5c680 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df81c885ec934e67b7288278ccc5c680. Latest consensus state: current_term: 1 leader_uuid: "df81c885ec934e67b7288278ccc5c680" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df81c885ec934e67b7288278ccc5c680" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43157 } } }
I20260619 08:19:04.494879 7180 sys_catalog.cc:458] T 00000000000000000000000000000000 P df81c885ec934e67b7288278ccc5c680 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:04.495628 7189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:19:04.496172 7189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:19:04.496626 7182 sys_catalog.cc:455] T 00000000000000000000000000000000 P df81c885ec934e67b7288278ccc5c680 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df81c885ec934e67b7288278ccc5c680" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df81c885ec934e67b7288278ccc5c680" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43157 } } }
I20260619 08:19:04.496976 7182 sys_catalog.cc:458] T 00000000000000000000000000000000 P df81c885ec934e67b7288278ccc5c680 [sys.catalog]: This master's current role is: LEADER
Jun 19 08:19:04 dist-test-slave-d3n8 krb5kdc[7094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:04.498832 7189 catalog_manager.cc:1347] Generated new cluster ID: c00034777d7d44c3a7a88f6b6aecf63b
I20260619 08:19:04.498881 7189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:19:04.503422 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:04.491212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43374 (local address 127.4.151.190:43157)
0619 08:19:04.491390 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:04.491397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:04.491421 (+ 24us) server_negotiation.cc:407] Connection header received
0619 08:19:04.492229 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:04.492237 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:04.492289 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:04.492461 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:04.493266 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:04.494485 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:04.495629 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:04.496145 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:04.496273 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:04.499186 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:04.499190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:04.499190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:04.499202 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:04.501198 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:04.501668 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:04.501672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:04.501672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:04.501790 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:04.501994 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:04.501995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:04.501995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:04.502300 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:04.502452 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:04.503006 (+ 554us) server_negotiation.cc:299] Negotiation successful
0619 08:19:04.503153 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":92,"threads_started":1}
I20260619 08:19:04.514923 7189 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:19:04.515596 7189 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:19:04.519526 7189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df81c885ec934e67b7288278ccc5c680: Generated new TSK 0
I20260619 08:19:04.519722 7189 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.151.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
I20260619 08:19:04.549021 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43157
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:04.632565 7201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:04.632790 7201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:04.632817 7201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:04.634140 7201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:04.634199 7201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:04.634228 7201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:19:04.635649 7201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:04.635867 7201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:04.636067 7201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:19:04.641685 7201 server_base.cc:1034] running on GCE node
W20260619 08:19:04.641637 7206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:04.641665 7207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:04.641662 7209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:04.642138 7201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:04.642385 7201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:04.643546 7201 hybrid_clock.cc:648] HybridClock initialized: now 1781857144643535 us; error 41 us; skew 500 ppm
Jun 19 08:19:04 dist-test-slave-d3n8 krb5kdc[7094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857144, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:04.646327 7201 init.cc:377] Logged in from keytab as kudu/127.4.151.129@KRBTEST.COM (short username kudu)
I20260619 08:19:04.646802 7201 webserver.cc:460] Webserver started at http://127.4.151.129:38267/ using document root <none> and password file <none>
I20260619 08:19:04.646996 7201 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:04.647037 7201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:04.647123 7201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:04.648137 7201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "1b728f61864244d5a1e54188fce5097f"
format_stamp: "Formatted at 2026-06-19 08:19:04 on dist-test-slave-d3n8"
server_key: "54083e96b597e819cf974588ad02a9d0"
server_key_iv: "0466d12319972bb6719f9ea665807217"
server_key_version: "encryptionkey@0"
I20260619 08:19:04.648442 7201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "1b728f61864244d5a1e54188fce5097f"
format_stamp: "Formatted at 2026-06-19 08:19:04 on dist-test-slave-d3n8"
server_key: "54083e96b597e819cf974588ad02a9d0"
server_key_iv: "0466d12319972bb6719f9ea665807217"
server_key_version: "encryptionkey@0"
I20260619 08:19:04.649765 7201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:19:04.650508 7216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:04.650694 7201 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:19:04.650767 7201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "1b728f61864244d5a1e54188fce5097f"
format_stamp: "Formatted at 2026-06-19 08:19:04 on dist-test-slave-d3n8"
server_key: "54083e96b597e819cf974588ad02a9d0"
server_key_iv: "0466d12319972bb6719f9ea665807217"
server_key_version: "encryptionkey@0"
I20260619 08:19:04.650823 7201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:04.662513 7201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:04.664911 7201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:04.665043 7201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:04.665242 7201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:04.665630 7201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:04.665688 7201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:04.665727 7201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:04.665752 7201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:04.671658 7201 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:33891
I20260619 08:19:04.671698 7329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:33891 every 8 connection(s)
I20260619 08:19:04.672019 7201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
I20260619 08:19:04.674069 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 7201
I20260619 08:19:04.674177 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:19:04.674345 4702 external_mini_cluster.cc:1428] Setting key 7e2214bc9fbdc233e5bd6fa2872883fa
Jun 19 08:19:04 dist-test-slave-d3n8 krb5kdc[7094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857144, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.151.130@KRBTEST.COM; defaulting to no policy
I20260619 08:19:04.682981 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:04.672714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:47285 (local address 127.4.151.190:43157)
0619 08:19:04.672784 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:04.672786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:04.673096 (+ 310us) server_negotiation.cc:407] Connection header received
0619 08:19:04.673697 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:04.673698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:04.673712 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:04.673767 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:04.674850 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:04.675417 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:04.676889 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:04.677245 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:04.677326 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:04.679752 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:04.679754 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:04.679755 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:04.679764 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:04.681479 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:04.681886 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:04.681888 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:04.681888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:04.681947 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:04.682202 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:04.682203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:04.682203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:04.682356 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:04.682442 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:04.682760 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 08:19:04.682850 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 08:19:04.683241 7332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:04.672732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:43157 (local address 127.4.151.129:47285)
0619 08:19:04.672890 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:04.673001 (+ 111us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:04.673569 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:04.673823 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:04.673826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:04.674561 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:04.674764 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:04.674768 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:04.675519 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:04.675520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:04.676798 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:04.676801 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:04.677338 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:04.677339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:04.677529 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:04.678132 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:04.678141 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:04.679642 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:04.681563 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:04.681570 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:04.681572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:04.681796 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:04.682021 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:04.682022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:04.682022 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:04.682130 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:04.682473 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:04.682482 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:04.682613 (+ 131us) client_negotiation.cc:769] Sending connection context
0619 08:19:04.682716 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 08:19:04.682946 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260619 08:19:04.683866 7330 heartbeater.cc:347] Connected to a master server at 127.4.151.190:43157
I20260619 08:19:04.684005 7330 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:04.684187 7330 heartbeater.cc:511] Master 127.4.151.190:43157 requested a full tablet report, sending...
I20260619 08:19:04.684736 7139 ts_manager.cc:194] Registered new tserver with Master: 1b728f61864244d5a1e54188fce5097f (127.4.151.129:33891)
I20260619 08:19:04.685521 7139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.129@KRBTEST.COM'} at 127.4.151.129:47285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
I20260619 08:19:04.704882 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43157
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:04.788702 7335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:04.788968 7335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:04.789016 7335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:04.790272 7335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:04.790347 7335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:04.790395 7335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:19:04.791878 7335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:04.792176 7335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:04.792423 7335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:04.798197 7341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:04.798285 7335 server_base.cc:1034] running on GCE node
W20260619 08:19:04.798229 7340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:04.798231 7343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:04.798754 7335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:04.799044 7335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:04.800238 7335 hybrid_clock.cc:648] HybridClock initialized: now 1781857144800222 us; error 49 us; skew 500 ppm
Jun 19 08:19:04 dist-test-slave-d3n8 krb5kdc[7094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857144, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:04.803031 7335 init.cc:377] Logged in from keytab as kudu/127.4.151.130@KRBTEST.COM (short username kudu)
I20260619 08:19:04.803522 7335 webserver.cc:460] Webserver started at http://127.4.151.130:36599/ using document root <none> and password file <none>
I20260619 08:19:04.803730 7335 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:04.803771 7335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:04.803864 7335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:04.804939 7335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "1bb47f32ae774bdf803e6010ddc49974"
format_stamp: "Formatted at 2026-06-19 08:19:04 on dist-test-slave-d3n8"
server_key: "23f2c218cd4d4cbb51710e50e4fe10d8"
server_key_iv: "1fa8ac21df4156782df007ec409c7d1a"
server_key_version: "encryptionkey@0"
I20260619 08:19:04.805538 7335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "1bb47f32ae774bdf803e6010ddc49974"
format_stamp: "Formatted at 2026-06-19 08:19:04 on dist-test-slave-d3n8"
server_key: "23f2c218cd4d4cbb51710e50e4fe10d8"
server_key_iv: "1fa8ac21df4156782df007ec409c7d1a"
server_key_version: "encryptionkey@0"
I20260619 08:19:04.806919 7335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:04.807709 7350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:04.808123 7335 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:04.808208 7335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "1bb47f32ae774bdf803e6010ddc49974"
format_stamp: "Formatted at 2026-06-19 08:19:04 on dist-test-slave-d3n8"
server_key: "23f2c218cd4d4cbb51710e50e4fe10d8"
server_key_iv: "1fa8ac21df4156782df007ec409c7d1a"
server_key_version: "encryptionkey@0"
I20260619 08:19:04.808398 7335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:04.836460 7335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:04.839071 7335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:04.839191 7335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:04.839735 7335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:04.840255 7335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:04.840303 7335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:04.840329 7335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:04.840344 7335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:04.846232 7335 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:40433
I20260619 08:19:04.846274 7463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:40433 every 8 connection(s)
I20260619 08:19:04.846674 7335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
I20260619 08:19:04.850471 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 7335
I20260619 08:19:04.850574 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:19:04.850787 4702 external_mini_cluster.cc:1428] Setting key 09d8e832e76766917b5b247aced43af2
Jun 19 08:19:04 dist-test-slave-d3n8 krb5kdc[7094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857144, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:04.858234 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:04.847679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:47359 (local address 127.4.151.190:43157)
0619 08:19:04.847729 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:04.847731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:04.847930 (+ 199us) server_negotiation.cc:407] Connection header received
0619 08:19:04.848442 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:04.848443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:04.848453 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:04.848522 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:04.849507 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:04.850036 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:04.851422 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:04.851713 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:04.851774 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:04.854625 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:04.854627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:04.854627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:04.854636 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:04.856626 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:04.857122 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:04.857123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:04.857124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:04.857189 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:04.857519 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:04.857520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:04.857520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:04.857664 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:04.857741 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:04.858046 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 08:19:04.858141 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:19:04.858431 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:04.847613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:43157 (local address 127.4.151.130:47359)
0619 08:19:04.847800 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:04.847842 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:04.848346 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:04.848582 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:04.848585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:04.849155 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:04.849352 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:04.849357 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:04.850133 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:04.850134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:04.851332 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:04.851335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:04.851830 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:04.851832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:04.852005 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:04.852590 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:04.852601 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:04.854495 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:04.856747 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:04.856756 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:04.856758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:04.857015 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:04.857259 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:04.857259 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:04.857260 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:04.857441 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:04.857742 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:04.857750 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:04.857899 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 08:19:04.858004 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 08:19:04.858173 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":115,"threads_started":1}
I20260619 08:19:04.859138 7464 heartbeater.cc:347] Connected to a master server at 127.4.151.190:43157
I20260619 08:19:04.859265 7464 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:04.859510 7464 heartbeater.cc:511] Master 127.4.151.190:43157 requested a full tablet report, sending...
I20260619 08:19:04.859963 7139 ts_manager.cc:194] Registered new tserver with Master: 1bb47f32ae774bdf803e6010ddc49974 (127.4.151.130:40433)
WARNING: no policy specified for kudu/127.4.151.131@KRBTEST.COM; defaulting to no policy
I20260619 08:19:04.860437 7139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.130@KRBTEST.COM'} at 127.4.151.130:47359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
I20260619 08:19:04.881295 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43157
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:04.966945 7469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:04.967211 7469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:04.967252 7469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:04.968490 7469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:04.968575 7469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:04.968619 7469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:19:04.970208 7469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:04.970508 7469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:04.970762 7469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:04.976264 7477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:04.976284 7475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:04.976298 7474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:04.976581 7469 server_base.cc:1034] running on GCE node
I20260619 08:19:04.976830 7469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:04.977113 7469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:04.978292 7469 hybrid_clock.cc:648] HybridClock initialized: now 1781857144978274 us; error 27 us; skew 500 ppm
Jun 19 08:19:04 dist-test-slave-d3n8 krb5kdc[7094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857144, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:04.980929 7469 init.cc:377] Logged in from keytab as kudu/127.4.151.131@KRBTEST.COM (short username kudu)
I20260619 08:19:04.981447 7469 webserver.cc:460] Webserver started at http://127.4.151.131:40649/ using document root <none> and password file <none>
I20260619 08:19:04.981685 7469 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:04.981752 7469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:04.981896 7469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:04.983034 7469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "f29d9158cff04749b8a297207d3f036b"
format_stamp: "Formatted at 2026-06-19 08:19:04 on dist-test-slave-d3n8"
server_key: "2871135761bae0f2b70762126b7774cd"
server_key_iv: "b4878024cc4210eb0e1f908897fd26ca"
server_key_version: "encryptionkey@0"
I20260619 08:19:04.983405 7469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "f29d9158cff04749b8a297207d3f036b"
format_stamp: "Formatted at 2026-06-19 08:19:04 on dist-test-slave-d3n8"
server_key: "2871135761bae0f2b70762126b7774cd"
server_key_iv: "b4878024cc4210eb0e1f908897fd26ca"
server_key_version: "encryptionkey@0"
I20260619 08:19:04.984752 7469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:19:04.985584 7484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:04.985817 7469 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:04.985927 7469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "f29d9158cff04749b8a297207d3f036b"
format_stamp: "Formatted at 2026-06-19 08:19:04 on dist-test-slave-d3n8"
server_key: "2871135761bae0f2b70762126b7774cd"
server_key_iv: "b4878024cc4210eb0e1f908897fd26ca"
server_key_version: "encryptionkey@0"
I20260619 08:19:04.986001 7469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:05.002686 7469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:05.005070 7469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:05.005234 7469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:05.005543 7469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:05.005872 7469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:05.005929 7469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:05.005973 7469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:05.006001 7469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:05.012319 7469 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:34439
I20260619 08:19:05.012362 7597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:34439 every 8 connection(s)
I20260619 08:19:05.012748 7469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
I20260619 08:19:05.016670 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 7469
I20260619 08:19:05.016791 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:19:05.017014 4702 external_mini_cluster.cc:1428] Setting key 025b397d4b90cad89d2d4838415d5ee7
Jun 19 08:19:05 dist-test-slave-d3n8 krb5kdc[7094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857144, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:05.023154 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:05.013554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:47083 (local address 127.4.151.190:43157)
0619 08:19:05.013624 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:05.013626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:05.013864 (+ 238us) server_negotiation.cc:407] Connection header received
0619 08:19:05.014524 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:05.014526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:05.014537 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:05.014614 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:05.015584 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:05.016103 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:05.017619 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:05.017914 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:05.017983 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:05.020402 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:05.020403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:05.020404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:05.020412 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:05.021810 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:05.022182 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:05.022183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:05.022184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:05.022237 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:05.022459 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:05.022460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:05.022460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:05.022566 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:05.022622 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:05.022962 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 08:19:05.023044 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 08:19:05.023263 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:05.013522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:43157 (local address 127.4.151.131:47083)
0619 08:19:05.013707 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:05.013751 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:05.014392 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:05.014673 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:05.014677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:05.015273 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:05.015470 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:05.015475 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:05.016197 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:05.016198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:05.017534 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:05.017537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:05.018019 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:05.018020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:05.018173 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:05.018661 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:05.018669 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:05.020175 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:05.021886 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:05.021891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:05.021893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:05.022112 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:05.022302 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:05.022302 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:05.022302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:05.022397 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:05.022649 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:05.022655 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:05.022784 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 08:19:05.022909 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 08:19:05.023049 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":113,"threads_started":1}
I20260619 08:19:05.023767 7598 heartbeater.cc:347] Connected to a master server at 127.4.151.190:43157
I20260619 08:19:05.023891 7598 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:05.024075 7598 heartbeater.cc:511] Master 127.4.151.190:43157 requested a full tablet report, sending...
I20260619 08:19:05.024526 7139 ts_manager.cc:194] Registered new tserver with Master: f29d9158cff04749b8a297207d3f036b (127.4.151.131:34439)
I20260619 08:19:05.024905 7139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.131@KRBTEST.COM'} at 127.4.151.131:47083
I20260619 08:19:05.029345 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 08:19:05.040918 7179 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:05.040029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43384 (local address 127.4.151.190:43157)
0619 08:19:05.040101 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:05.040103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:05.040179 (+ 76us) server_negotiation.cc:407] Connection header received
0619 08:19:05.040235 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:05.040235 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:05.040245 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:05.040308 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:05.040823 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43384: BlockingRecv error: recv got EOF from 127.0.0.1:43384 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 08:19:05.041834 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 7201
I20260619 08:19:05.046692 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 7335
I20260619 08:19:05.050853 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 7469
I20260619 08:19:05.055150 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 7107
2026-06-19T08:19:05Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1869 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:19:05 dist-test-slave-d3n8 krb5kdc[7610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:19:05 dist-test-slave-d3n8 krb5kdc[7610](info): set up 2 sockets
Jun 19 08:19:05 dist-test-slave-d3n8 krb5kdc[7610](info): commencing operation
krb5kdc: starting...
W20260619 08:19:06.109323 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:19:06 dist-test-slave-d3n8 krb5kdc[7610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:19:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:19:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:19:06.242681 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:41537
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:46451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:41537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:06.326789 7623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:06.327004 7623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:06.327028 7623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:06.328292 7623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:19:06.328343 7623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:06.328357 7623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:06.328372 7623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:19:06.328383 7623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:19:06.329993 7623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:46451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:41537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:41537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:06.330246 7623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:06.330498 7623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:06.335801 7629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:06.335807 7628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:06.335839 7631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:06.336087 7623 server_base.cc:1034] running on GCE node
I20260619 08:19:06.336357 7623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:06.336720 7623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:06.337913 7623 hybrid_clock.cc:648] HybridClock initialized: now 1781857146337874 us; error 76 us; skew 500 ppm
Jun 19 08:19:06 dist-test-slave-d3n8 krb5kdc[7610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857146, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:06.340649 7623 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:19:06.341252 7623 webserver.cc:460] Webserver started at http://127.4.151.190:36665/ using document root <none> and password file <none>
I20260619 08:19:06.341573 7623 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:06.341647 7623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:06.341814 7623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:06.342957 7623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "23c547d381b745ec8caa446287a60539"
format_stamp: "Formatted at 2026-06-19 08:19:06 on dist-test-slave-d3n8"
server_key: "3a57c3a069839fe6d2e7a91d8f5d0065"
server_key_iv: "b55c9e9d87d875ec55c944880a47d389"
server_key_version: "encryptionkey@0"
I20260619 08:19:06.343349 7623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "23c547d381b745ec8caa446287a60539"
format_stamp: "Formatted at 2026-06-19 08:19:06 on dist-test-slave-d3n8"
server_key: "3a57c3a069839fe6d2e7a91d8f5d0065"
server_key_iv: "b55c9e9d87d875ec55c944880a47d389"
server_key_version: "encryptionkey@0"
I20260619 08:19:06.344769 7623 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:19:06.345748 7638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:06.346022 7623 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:06.346134 7623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "23c547d381b745ec8caa446287a60539"
format_stamp: "Formatted at 2026-06-19 08:19:06 on dist-test-slave-d3n8"
server_key: "3a57c3a069839fe6d2e7a91d8f5d0065"
server_key_iv: "b55c9e9d87d875ec55c944880a47d389"
server_key_version: "encryptionkey@0"
I20260619 08:19:06.346211 7623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:06.360338 7623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:06.362862 7623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:06.363037 7623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:06.367282 7623 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:41537
I20260619 08:19:06.367323 7690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:41537 every 8 connection(s)
I20260619 08:19:06.367689 7623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:19:06.368044 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 7623
I20260619 08:19:06.368160 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:19:06.368358 4702 external_mini_cluster.cc:1428] Setting key 107de98a43a9b5ccf8cd8337a5772a4f
I20260619 08:19:06.369073 7691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:06.372400 7691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23c547d381b745ec8caa446287a60539: Bootstrap starting.
I20260619 08:19:06.373039 7691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23c547d381b745ec8caa446287a60539: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:06.373344 7691 log.cc:824] T 00000000000000000000000000000000 P 23c547d381b745ec8caa446287a60539: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:06.374150 7691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23c547d381b745ec8caa446287a60539: No bootstrap required, opened a new log
Jun 19 08:19:06 dist-test-slave-d3n8 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:06.375975 7691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23c547d381b745ec8caa446287a60539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c547d381b745ec8caa446287a60539" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 41537 } }
I20260619 08:19:06.376197 7691 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23c547d381b745ec8caa446287a60539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:06.376256 7691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23c547d381b745ec8caa446287a60539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c547d381b745ec8caa446287a60539, State: Initialized, Role: FOLLOWER
I20260619 08:19:06.376389 7691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23c547d381b745ec8caa446287a60539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c547d381b745ec8caa446287a60539" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 41537 } }
I20260619 08:19:06.376477 7691 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23c547d381b745ec8caa446287a60539 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:19:06.376518 7691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23c547d381b745ec8caa446287a60539 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:19:06.376559 7691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23c547d381b745ec8caa446287a60539 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:06.377244 7691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23c547d381b745ec8caa446287a60539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c547d381b745ec8caa446287a60539" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 41537 } }
I20260619 08:19:06.377367 7691 leader_election.cc:302] T 00000000000000000000000000000000 P 23c547d381b745ec8caa446287a60539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23c547d381b745ec8caa446287a60539; no voters:
I20260619 08:19:06.377591 7691 leader_election.cc:288] T 00000000000000000000000000000000 P 23c547d381b745ec8caa446287a60539 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:19:06.377779 7697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23c547d381b745ec8caa446287a60539 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:06.377873 7691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23c547d381b745ec8caa446287a60539 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:19:06.378003 7697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23c547d381b745ec8caa446287a60539 [term 1 LEADER]: Becoming Leader. State: Replica: 23c547d381b745ec8caa446287a60539, State: Running, Role: LEADER
I20260619 08:19:06.378152 7697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23c547d381b745ec8caa446287a60539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c547d381b745ec8caa446287a60539" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 41537 } }
I20260619 08:19:06.378574 7698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23c547d381b745ec8caa446287a60539 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23c547d381b745ec8caa446287a60539. Latest consensus state: current_term: 1 leader_uuid: "23c547d381b745ec8caa446287a60539" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c547d381b745ec8caa446287a60539" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 41537 } } }
I20260619 08:19:06.378671 7698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23c547d381b745ec8caa446287a60539 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:06.379038 7696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23c547d381b745ec8caa446287a60539 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23c547d381b745ec8caa446287a60539" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c547d381b745ec8caa446287a60539" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 41537 } } }
I20260619 08:19:06.379108 7696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23c547d381b745ec8caa446287a60539 [sys.catalog]: This master's current role is: LEADER
W20260619 08:19:06.379702 7711 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 23c547d381b745ec8caa446287a60539: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:19:06.379781 7711 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:19:06.379868 7705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:19:06.380074 7705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:19:06.380807 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:06.368934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49004 (local address 127.4.151.190:41537)
0619 08:19:06.369298 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:06.369306 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:06.369337 (+ 31us) server_negotiation.cc:407] Connection header received
0619 08:19:06.369911 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:06.369920 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:06.369986 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:06.370141 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:06.370805 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:06.372220 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:06.373188 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:06.373774 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:06.373942 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:06.376570 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:06.376575 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:06.376575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:06.376588 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:06.378898 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:06.379365 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:06.379369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:06.379369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:06.379478 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:06.379702 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:06.379702 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:06.379703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:06.380127 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:06.380297 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:06.380477 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 08:19:06.380613 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":134,"threads_started":1}
I20260619 08:19:06.381757 7705 catalog_manager.cc:1347] Generated new cluster ID: 398bd0ffa9754f80b13608b095b80285
I20260619 08:19:06.381752 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:19:06.381814 7705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:19:06.395653 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:06.392071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49018 (local address 127.4.151.190:41537)
0619 08:19:06.392119 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:06.392121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:06.392246 (+ 125us) server_negotiation.cc:407] Connection header received
0619 08:19:06.392354 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:06.392355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:06.392367 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:06.392419 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:06.392812 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:06.393312 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:06.394097 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:06.394338 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:06.394419 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:06.395158 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:06.395159 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:06.395159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:19:06.395362 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:06.395378 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:19:06.395384 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:06.395453 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:06.395519 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 08:19:06.395542 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 08:19:06.407728 7705 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:19:06.408349 7705 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:19:06.435802 7705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23c547d381b745ec8caa446287a60539: Generated new TSK 0
I20260619 08:19:06.436213 7705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:19:06.448271 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 7623
2026-06-19T08:19:06Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1391 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:19:06 dist-test-slave-d3n8 krb5kdc[7724](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:19:06 dist-test-slave-d3n8 krb5kdc[7724](info): set up 2 sockets
Jun 19 08:19:06 dist-test-slave-d3n8 krb5kdc[7724](info): commencing operation
krb5kdc: starting...
W20260619 08:19:07.558610 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:19:07 dist-test-slave-d3n8 krb5kdc[7724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:19:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:19:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:19:07.685076 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:38611
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:46171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:38611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:07.771596 7737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:07.771811 7737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:07.771834 7737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:07.772933 7737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:19:07.772975 7737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:07.772989 7737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:07.773002 7737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:19:07.773015 7737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:19:07.774572 7737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:46171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:38611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:38611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:07.774788 7737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:07.774966 7737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:07.780274 7743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:07.780299 7745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:07.780277 7742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:07.780640 7737 server_base.cc:1034] running on GCE node
I20260619 08:19:07.780903 7737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:07.781208 7737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:07.782392 7737 hybrid_clock.cc:648] HybridClock initialized: now 1781857147782374 us; error 58 us; skew 500 ppm
Jun 19 08:19:07 dist-test-slave-d3n8 krb5kdc[7724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857147, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:07.785156 7737 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:19:07.785640 7737 webserver.cc:460] Webserver started at http://127.4.151.190:45405/ using document root <none> and password file <none>
I20260619 08:19:07.785878 7737 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:07.785924 7737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:07.786069 7737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:07.787166 7737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "2230bc7b61654392b613aff2ec654a12"
format_stamp: "Formatted at 2026-06-19 08:19:07 on dist-test-slave-d3n8"
server_key: "7aecc18320d9666545a856e53e2eaa90"
server_key_iv: "67ba923aee25b4ff60a832ea0b7eedd2"
server_key_version: "encryptionkey@0"
I20260619 08:19:07.787540 7737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "2230bc7b61654392b613aff2ec654a12"
format_stamp: "Formatted at 2026-06-19 08:19:07 on dist-test-slave-d3n8"
server_key: "7aecc18320d9666545a856e53e2eaa90"
server_key_iv: "67ba923aee25b4ff60a832ea0b7eedd2"
server_key_version: "encryptionkey@0"
I20260619 08:19:07.789036 7737 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:19:07.790053 7752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:07.790300 7737 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 08:19:07.790397 7737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "2230bc7b61654392b613aff2ec654a12"
format_stamp: "Formatted at 2026-06-19 08:19:07 on dist-test-slave-d3n8"
server_key: "7aecc18320d9666545a856e53e2eaa90"
server_key_iv: "67ba923aee25b4ff60a832ea0b7eedd2"
server_key_version: "encryptionkey@0"
I20260619 08:19:07.790462 7737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:07.808276 7737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:07.810760 7737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:07.810882 7737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:07.814976 7737 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:38611
I20260619 08:19:07.815127 7804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:38611 every 8 connection(s)
I20260619 08:19:07.815430 7737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:19:07.816509 7805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:07.818926 7805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2230bc7b61654392b613aff2ec654a12: Bootstrap starting.
I20260619 08:19:07.819552 7805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2230bc7b61654392b613aff2ec654a12: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:07.819850 7805 log.cc:824] T 00000000000000000000000000000000 P 2230bc7b61654392b613aff2ec654a12: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:07.820617 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 7737
I20260619 08:19:07.820636 7805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2230bc7b61654392b613aff2ec654a12: No bootstrap required, opened a new log
I20260619 08:19:07.820694 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:19:07.820996 4702 external_mini_cluster.cc:1428] Setting key 50c6eba90af34c4f6f827ccf140480ba
I20260619 08:19:07.823021 7805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2230bc7b61654392b613aff2ec654a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2230bc7b61654392b613aff2ec654a12" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38611 } }
I20260619 08:19:07.823355 7805 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2230bc7b61654392b613aff2ec654a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:07.823391 7805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2230bc7b61654392b613aff2ec654a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2230bc7b61654392b613aff2ec654a12, State: Initialized, Role: FOLLOWER
I20260619 08:19:07.823537 7805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2230bc7b61654392b613aff2ec654a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2230bc7b61654392b613aff2ec654a12" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38611 } }
I20260619 08:19:07.823797 7805 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2230bc7b61654392b613aff2ec654a12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:19:07.823844 7805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2230bc7b61654392b613aff2ec654a12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:19:07.823877 7805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2230bc7b61654392b613aff2ec654a12 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:07.824595 7805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2230bc7b61654392b613aff2ec654a12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2230bc7b61654392b613aff2ec654a12" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38611 } }
I20260619 08:19:07.824703 7805 leader_election.cc:302] T 00000000000000000000000000000000 P 2230bc7b61654392b613aff2ec654a12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2230bc7b61654392b613aff2ec654a12; no voters:
I20260619 08:19:07.824865 7805 leader_election.cc:288] T 00000000000000000000000000000000 P 2230bc7b61654392b613aff2ec654a12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:19:07.825017 7810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2230bc7b61654392b613aff2ec654a12 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:07.825131 7805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2230bc7b61654392b613aff2ec654a12 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:19:07.825234 7810 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2230bc7b61654392b613aff2ec654a12 [term 1 LEADER]: Becoming Leader. State: Replica: 2230bc7b61654392b613aff2ec654a12, State: Running, Role: LEADER
I20260619 08:19:07.825368 7810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2230bc7b61654392b613aff2ec654a12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2230bc7b61654392b613aff2ec654a12" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38611 } }
I20260619 08:19:07.825856 7812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2230bc7b61654392b613aff2ec654a12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2230bc7b61654392b613aff2ec654a12. Latest consensus state: current_term: 1 leader_uuid: "2230bc7b61654392b613aff2ec654a12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2230bc7b61654392b613aff2ec654a12" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38611 } } }
I20260619 08:19:07.825830 7811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2230bc7b61654392b613aff2ec654a12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2230bc7b61654392b613aff2ec654a12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2230bc7b61654392b613aff2ec654a12" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38611 } } }
I20260619 08:19:07.825927 7812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2230bc7b61654392b613aff2ec654a12 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:07.825932 7811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2230bc7b61654392b613aff2ec654a12 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:07.826128 7819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:19:07.826696 7819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:19:07.828792 7819 catalog_manager.cc:1347] Generated new cluster ID: 17b31473d5bc4a0596997ee70c426ec9
I20260619 08:19:07.828848 7819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:19:07 dist-test-slave-d3n8 krb5kdc[7724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:07.834779 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:07.821643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56978 (local address 127.4.151.190:38611)
0619 08:19:07.821846 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:07.821855 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:07.821901 (+ 46us) server_negotiation.cc:407] Connection header received
0619 08:19:07.823285 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:07.823295 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:07.823362 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:07.823549 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:07.824394 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:07.825858 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:07.827207 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:07.827704 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:07.827833 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:07.830918 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:07.830923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:07.830923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:07.830937 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:07.833036 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:07.833463 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:07.833467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:07.833468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:07.833598 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:07.833815 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:07.833816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:07.833816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:07.834164 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:07.834328 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:07.834421 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 08:19:07.834568 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":89,"threads_started":1}
I20260619 08:19:07.842480 7819 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:19:07.843099 7819 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:19:07.848688 7819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2230bc7b61654392b613aff2ec654a12: Generated new TSK 0
I20260619 08:19:07.848907 7819 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.151.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
I20260619 08:19:07.885581 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:38611
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:46171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:07.970942 7831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:07.971189 7831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:07.971227 7831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:07.972411 7831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:07.972483 7831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:07.972530 7831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:19:07.974139 7831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:46171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:38611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:07.974407 7831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:07.974650 7831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:19:07.980456 7831 server_base.cc:1034] running on GCE node
W20260619 08:19:07.980417 7839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:07.980427 7836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:07.980427 7837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:07.980928 7831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:07.981323 7831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:07.982520 7831 hybrid_clock.cc:648] HybridClock initialized: now 1781857147982493 us; error 43 us; skew 500 ppm
Jun 19 08:19:07 dist-test-slave-d3n8 krb5kdc[7724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857147, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:07.985317 7831 init.cc:377] Logged in from keytab as kudu/127.4.151.129@KRBTEST.COM (short username kudu)
I20260619 08:19:07.985813 7831 webserver.cc:460] Webserver started at http://127.4.151.129:33193/ using document root <none> and password file <none>
I20260619 08:19:07.986070 7831 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:07.986116 7831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:07.986286 7831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:07.987357 7831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "2e52b5fa99344a088ded735a1b4b2bea"
format_stamp: "Formatted at 2026-06-19 08:19:07 on dist-test-slave-d3n8"
server_key: "4b8be55d6f1db76d50134330a834adee"
server_key_iv: "d21270cedb7b8b5e3ea0a51ff0c8a647"
server_key_version: "encryptionkey@0"
I20260619 08:19:07.987725 7831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "2e52b5fa99344a088ded735a1b4b2bea"
format_stamp: "Formatted at 2026-06-19 08:19:07 on dist-test-slave-d3n8"
server_key: "4b8be55d6f1db76d50134330a834adee"
server_key_iv: "d21270cedb7b8b5e3ea0a51ff0c8a647"
server_key_version: "encryptionkey@0"
I20260619 08:19:07.989156 7831 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:19:07.990041 7846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:07.990329 7831 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:07.990411 7831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "2e52b5fa99344a088ded735a1b4b2bea"
format_stamp: "Formatted at 2026-06-19 08:19:07 on dist-test-slave-d3n8"
server_key: "4b8be55d6f1db76d50134330a834adee"
server_key_iv: "d21270cedb7b8b5e3ea0a51ff0c8a647"
server_key_version: "encryptionkey@0"
I20260619 08:19:07.990562 7831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:08.012523 7831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:08.015169 7831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:08.015295 7831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:08.015599 7831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:08.015957 7831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:08.015992 7831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:08.016016 7831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:08.016028 7831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:08.022294 7831 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:36117
I20260619 08:19:08.022329 7959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:36117 every 8 connection(s)
I20260619 08:19:08.022702 7831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
Jun 19 08:19:08 dist-test-slave-d3n8 krb5kdc[7724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857147, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:08.031602 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 7831
I20260619 08:19:08.031729 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:19:08.031922 4702 external_mini_cluster.cc:1428] Setting key 61a1cf7745379d477a39691a821e87c4
I20260619 08:19:08.034276 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:08.023577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:35731 (local address 127.4.151.190:38611)
0619 08:19:08.023641 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:08.023643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:08.023786 (+ 143us) server_negotiation.cc:407] Connection header received
0619 08:19:08.024456 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:08.024457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:08.024470 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:08.024548 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:08.025783 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:08.026332 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:08.027451 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:08.027668 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:08.027740 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:08.030545 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:08.030547 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:08.030547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:08.030563 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:08.032678 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:08.033111 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:08.033112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:08.033113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:08.033179 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:08.033509 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:08.033510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:08.033510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:08.033638 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:08.033718 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:08.034075 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 08:19:08.034192 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 08:19:08.034495 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:08.023479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:38611 (local address 127.4.151.129:35731)
0619 08:19:08.023653 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:08.023696 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:08.024330 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:08.024625 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:08.024628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:08.025411 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:08.025666 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:08.025672 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:08.026437 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:08.026438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:08.027348 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:08.027351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:08.027760 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:08.027761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:08.027882 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:08.028480 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:08.028488 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:08.030344 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:08.032757 (+ 2413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:08.032766 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:08.032769 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:08.033028 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:08.033254 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:08.033254 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:08.033255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:08.033417 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:08.033709 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:08.033716 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:08.033873 (+ 157us) client_negotiation.cc:769] Sending connection context
0619 08:19:08.033996 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 08:19:08.034203 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260619 08:19:08.035028 7960 heartbeater.cc:347] Connected to a master server at 127.4.151.190:38611
I20260619 08:19:08.035145 7960 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:08.035332 7960 heartbeater.cc:511] Master 127.4.151.190:38611 requested a full tablet report, sending...
I20260619 08:19:08.036041 7769 ts_manager.cc:194] Registered new tserver with Master: 2e52b5fa99344a088ded735a1b4b2bea (127.4.151.129:36117)
I20260619 08:19:08.036670 7769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.129@KRBTEST.COM'} at 127.4.151.129:35731
WARNING: no policy specified for kudu/127.4.151.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
I20260619 08:19:08.061088 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:38611
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:46171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:08.144629 7965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:08.144845 7965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:08.144876 7965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:08.146145 7965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:08.146195 7965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:08.146224 7965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:19:08.147758 7965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:46171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:38611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:08.148028 7965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:08.148283 7965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:08.153903 7971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:08.153903 7970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:08.154093 7965 server_base.cc:1034] running on GCE node
W20260619 08:19:08.154016 7973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:08.154525 7965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:08.154737 7965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:08.155932 7965 hybrid_clock.cc:648] HybridClock initialized: now 1781857148155886 us; error 59 us; skew 500 ppm
Jun 19 08:19:08 dist-test-slave-d3n8 krb5kdc[7724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857148, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:08.158581 7965 init.cc:377] Logged in from keytab as kudu/127.4.151.130@KRBTEST.COM (short username kudu)
I20260619 08:19:08.159015 7965 webserver.cc:460] Webserver started at http://127.4.151.130:38941/ using document root <none> and password file <none>
I20260619 08:19:08.159227 7965 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:08.159268 7965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:08.159355 7965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:08.160411 7965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "55d94155910f4d2f999d7c19ab05df46"
format_stamp: "Formatted at 2026-06-19 08:19:08 on dist-test-slave-d3n8"
server_key: "9b435a9cd5de58a990a7f8149941010c"
server_key_iv: "e3ecf1e4f52585258876d31ba4d609d7"
server_key_version: "encryptionkey@0"
I20260619 08:19:08.160732 7965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "55d94155910f4d2f999d7c19ab05df46"
format_stamp: "Formatted at 2026-06-19 08:19:08 on dist-test-slave-d3n8"
server_key: "9b435a9cd5de58a990a7f8149941010c"
server_key_iv: "e3ecf1e4f52585258876d31ba4d609d7"
server_key_version: "encryptionkey@0"
I20260619 08:19:08.162163 7965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 08:19:08.162994 7980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:08.163290 7965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:08.163374 7965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "55d94155910f4d2f999d7c19ab05df46"
format_stamp: "Formatted at 2026-06-19 08:19:08 on dist-test-slave-d3n8"
server_key: "9b435a9cd5de58a990a7f8149941010c"
server_key_iv: "e3ecf1e4f52585258876d31ba4d609d7"
server_key_version: "encryptionkey@0"
I20260619 08:19:08.163476 7965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:08.186384 7965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:08.188846 7965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:08.188966 7965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:08.189249 7965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:08.189596 7965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:08.189635 7965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:08.189657 7965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:08.189670 7965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:08.195853 7965 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:41129
I20260619 08:19:08.195945 8093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:41129 every 8 connection(s)
I20260619 08:19:08.196342 7965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
Jun 19 08:19:08 dist-test-slave-d3n8 krb5kdc[7724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857148, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:08.206503 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 7965
I20260619 08:19:08.206657 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:19:08.206811 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:08.197166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:60223 (local address 127.4.151.190:38611)
0619 08:19:08.197210 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:08.197213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:08.197481 (+ 268us) server_negotiation.cc:407] Connection header received
0619 08:19:08.198059 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:08.198061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:08.198154 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:08.198247 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:08.199280 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:08.199798 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:08.200860 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:08.201090 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:08.201170 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:08.203544 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:08.203546 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:08.203546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:08.203555 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:08.205197 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:08.205766 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:08.205767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:08.205768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:08.205832 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:08.206105 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:08.206106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:08.206106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:08.206234 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:08.206306 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:08.206612 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 08:19:08.206710 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:19:08.206874 4702 external_mini_cluster.cc:1428] Setting key b16970b6fff47283ba8dd23eb36b2b26
I20260619 08:19:08.206986 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:08.197120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:38611 (local address 127.4.151.130:60223)
0619 08:19:08.197288 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:08.197332 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:08.197942 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:08.198328 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:08.198332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:08.198970 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:08.199165 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:08.199169 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:08.199923 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:08.199924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:08.200762 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:08.200764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:08.201176 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:08.201177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:08.201312 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:08.201889 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:08.201897 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:08.203430 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:08.205341 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:08.205348 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:08.205350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:08.205661 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:08.205910 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:08.205911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:08.205911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:08.206030 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:08.206308 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:08.206315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:08.206464 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 08:19:08.206567 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 08:19:08.206736 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260619 08:19:08.207525 8094 heartbeater.cc:347] Connected to a master server at 127.4.151.190:38611
I20260619 08:19:08.207653 8094 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:08.207858 8094 heartbeater.cc:511] Master 127.4.151.190:38611 requested a full tablet report, sending...
I20260619 08:19:08.208288 7769 ts_manager.cc:194] Registered new tserver with Master: 55d94155910f4d2f999d7c19ab05df46 (127.4.151.130:41129)
I20260619 08:19:08.208828 7769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.130@KRBTEST.COM'} at 127.4.151.130:60223
WARNING: no policy specified for kudu/127.4.151.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
I20260619 08:19:08.234589 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:38611
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:46171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:08.317945 8099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:08.318152 8099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:08.318177 8099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:08.319291 8099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:08.319331 8099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:08.319375 8099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:19:08.320744 8099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:46171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:38611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:08.320930 8099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:08.321108 8099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:08.326822 8107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:08.326805 8104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:08.326851 8099 server_base.cc:1034] running on GCE node
W20260619 08:19:08.326794 8105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:08.327528 8099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:08.327765 8099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:08.328939 8099 hybrid_clock.cc:648] HybridClock initialized: now 1781857148328928 us; error 49 us; skew 500 ppm
Jun 19 08:19:08 dist-test-slave-d3n8 krb5kdc[7724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857148, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:08.331703 8099 init.cc:377] Logged in from keytab as kudu/127.4.151.131@KRBTEST.COM (short username kudu)
I20260619 08:19:08.332171 8099 webserver.cc:460] Webserver started at http://127.4.151.131:34713/ using document root <none> and password file <none>
I20260619 08:19:08.332405 8099 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:08.332495 8099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:08.332649 8099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:08.333770 8099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "9aae91bc91e543dbb756359ef5b3871f"
format_stamp: "Formatted at 2026-06-19 08:19:08 on dist-test-slave-d3n8"
server_key: "9b7580cd448c9bc8e0cb80b909e45869"
server_key_iv: "a59d49ed422c04656eeffa7be94d5041"
server_key_version: "encryptionkey@0"
I20260619 08:19:08.334141 8099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "9aae91bc91e543dbb756359ef5b3871f"
format_stamp: "Formatted at 2026-06-19 08:19:08 on dist-test-slave-d3n8"
server_key: "9b7580cd448c9bc8e0cb80b909e45869"
server_key_iv: "a59d49ed422c04656eeffa7be94d5041"
server_key_version: "encryptionkey@0"
I20260619 08:19:08.335481 8099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:19:08.336285 8114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:08.336469 8099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:08.336571 8099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "9aae91bc91e543dbb756359ef5b3871f"
format_stamp: "Formatted at 2026-06-19 08:19:08 on dist-test-slave-d3n8"
server_key: "9b7580cd448c9bc8e0cb80b909e45869"
server_key_iv: "a59d49ed422c04656eeffa7be94d5041"
server_key_version: "encryptionkey@0"
I20260619 08:19:08.336645 8099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:08.347488 8099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:08.349988 8099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:08.350158 8099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:08.350397 8099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:08.350721 8099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:08.350776 8099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:08.350818 8099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:08.350845 8099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:08.357110 8099 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:34939
I20260619 08:19:08.357151 8227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:34939 every 8 connection(s)
I20260619 08:19:08.357549 8099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
I20260619 08:19:08.359777 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 8099
I20260619 08:19:08.359989 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:19:08.360186 4702 external_mini_cluster.cc:1428] Setting key b15faae76ea6b1e2cae1aa9323ce7243
Jun 19 08:19:08 dist-test-slave-d3n8 krb5kdc[7724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857148, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:08.368441 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:08.358349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:37575 (local address 127.4.151.190:38611)
0619 08:19:08.358431 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:08.358434 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:08.358678 (+ 244us) server_negotiation.cc:407] Connection header received
0619 08:19:08.359206 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:08.359207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:08.359219 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:08.359344 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:08.360499 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:08.361322 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:08.362521 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:08.362729 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:08.362799 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:08.365237 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:08.365238 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:08.365239 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:08.365246 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:08.367022 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:08.367450 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:08.367451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:08.367452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:08.367510 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:08.367814 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:08.367814 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:08.367815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:08.367937 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:08.367996 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:08.368295 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 08:19:08.368366 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 08:19:08.368741 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:08.358321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:38611 (local address 127.4.151.131:37575)
0619 08:19:08.358530 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:08.358575 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:08.359080 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:08.359389 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:08.359392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:08.360108 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:08.360411 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:08.360424 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:08.361497 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:08.361497 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:08.362424 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:08.362427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:08.362820 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:08.362821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:08.362938 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:08.363553 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:08.363561 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:08.365139 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:08.367116 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:08.367122 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:08.367124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:08.367370 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:08.367624 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:08.367625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:08.367625 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:08.367747 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:08.368026 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:08.368032 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:08.368165 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 08:19:08.368267 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 08:19:08.368398 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":83,"threads_started":1}
I20260619 08:19:08.369453 8228 heartbeater.cc:347] Connected to a master server at 127.4.151.190:38611
I20260619 08:19:08.369568 8228 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:08.369794 8228 heartbeater.cc:511] Master 127.4.151.190:38611 requested a full tablet report, sending...
I20260619 08:19:08.370232 7769 ts_manager.cc:194] Registered new tserver with Master: 9aae91bc91e543dbb756359ef5b3871f (127.4.151.131:34939)
I20260619 08:19:08.370661 7769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.131@KRBTEST.COM'} at 127.4.151.131:37575
I20260619 08:19:08.372058 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 08:19:08 dist-test-slave-d3n8 krb5kdc[7724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857148, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 08:19:08 dist-test-slave-d3n8 krb5kdc[7724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857148, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:08.396793 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:08.389041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56984 (local address 127.4.151.190:38611)
0619 08:19:08.389112 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:08.389114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:08.389125 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:19:08.389160 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:08.389161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:08.389170 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:08.389235 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:08.389924 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:08.390433 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:08.391194 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:08.391386 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:08.391463 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:08.394247 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:08.394248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:08.394248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:08.394255 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:08.395643 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:08.395983 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:08.395984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:08.395984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:08.396036 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:08.396275 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:08.396276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:08.396276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:08.396421 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:08.396523 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:08.396663 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 08:19:08.396727 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 08:19:08.397040 7769 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56984
I20260619 08:19:08.397706 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 7831
I20260619 08:19:08.402545 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 7965
I20260619 08:19:08.407392 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 8099
I20260619 08:19:08.411840 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 7737
2026-06-19T08:19:08Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1969 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:19:08 dist-test-slave-d3n8 krb5kdc[8240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:19:08 dist-test-slave-d3n8 krb5kdc[8240](info): set up 2 sockets
Jun 19 08:19:08 dist-test-slave-d3n8 krb5kdc[8240](info): commencing operation
krb5kdc: starting...
W20260619 08:19:09.470335 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:19:09 dist-test-slave-d3n8 krb5kdc[8240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:19:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:19:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:19:09.602152 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:43463
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:43463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:09.685685 8253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:09.685897 8253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:09.685920 8253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:09.687090 8253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:19:09.687135 8253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:09.687150 8253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:09.687162 8253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:19:09.687175 8253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:19:09.688687 8253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:43463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:43463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:09.688938 8253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:09.689173 8253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:09.694219 8259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:09.694228 8258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:09.694233 8261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:09.694512 8253 server_base.cc:1034] running on GCE node
I20260619 08:19:09.694805 8253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:09.695184 8253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:09.696413 8253 hybrid_clock.cc:648] HybridClock initialized: now 1781857149696370 us; error 35 us; skew 500 ppm
Jun 19 08:19:09 dist-test-slave-d3n8 krb5kdc[8240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857149, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:09.699108 8253 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:19:09.699517 8253 webserver.cc:460] Webserver started at http://127.4.151.190:44801/ using document root <none> and password file <none>
I20260619 08:19:09.699702 8253 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:09.699741 8253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:09.699829 8253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:09.700848 8253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "e81f7c56da5946ac996d06b7c1638c45"
format_stamp: "Formatted at 2026-06-19 08:19:09 on dist-test-slave-d3n8"
server_key: "a9c7545357f1776c14fbaab526766ce8"
server_key_iv: "16561405bfbab18a2b6175f78f30e566"
server_key_version: "encryptionkey@0"
I20260619 08:19:09.701154 8253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "e81f7c56da5946ac996d06b7c1638c45"
format_stamp: "Formatted at 2026-06-19 08:19:09 on dist-test-slave-d3n8"
server_key: "a9c7545357f1776c14fbaab526766ce8"
server_key_iv: "16561405bfbab18a2b6175f78f30e566"
server_key_version: "encryptionkey@0"
I20260619 08:19:09.702570 8253 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:19:09.703395 8268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:09.703691 8253 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:09.703783 8253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "e81f7c56da5946ac996d06b7c1638c45"
format_stamp: "Formatted at 2026-06-19 08:19:09 on dist-test-slave-d3n8"
server_key: "a9c7545357f1776c14fbaab526766ce8"
server_key_iv: "16561405bfbab18a2b6175f78f30e566"
server_key_version: "encryptionkey@0"
I20260619 08:19:09.703857 8253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:09.724213 8253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:09.726470 8253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:09.726605 8253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:09.730530 8253 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:43463
I20260619 08:19:09.730612 8320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:43463 every 8 connection(s)
I20260619 08:19:09.731142 8253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:19:09.731923 8321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:09.734519 8321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e81f7c56da5946ac996d06b7c1638c45: Bootstrap starting.
I20260619 08:19:09.735183 8321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e81f7c56da5946ac996d06b7c1638c45: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:09.735500 8321 log.cc:824] T 00000000000000000000000000000000 P e81f7c56da5946ac996d06b7c1638c45: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:09.736560 8321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e81f7c56da5946ac996d06b7c1638c45: No bootstrap required, opened a new log
I20260619 08:19:09.737682 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 8253
I20260619 08:19:09.737807 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:19:09.737985 4702 external_mini_cluster.cc:1428] Setting key 83ed7e797ddb5d463ed1809f0c5c46c2
I20260619 08:19:09.738735 8321 raft_consensus.cc:348] T 00000000000000000000000000000000 P e81f7c56da5946ac996d06b7c1638c45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e81f7c56da5946ac996d06b7c1638c45" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43463 } }
I20260619 08:19:09.738889 8321 raft_consensus.cc:374] T 00000000000000000000000000000000 P e81f7c56da5946ac996d06b7c1638c45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:09.738916 8321 raft_consensus.cc:729] T 00000000000000000000000000000000 P e81f7c56da5946ac996d06b7c1638c45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e81f7c56da5946ac996d06b7c1638c45, State: Initialized, Role: FOLLOWER
I20260619 08:19:09.739078 8321 consensus_queue.cc:260] T 00000000000000000000000000000000 P e81f7c56da5946ac996d06b7c1638c45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e81f7c56da5946ac996d06b7c1638c45" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43463 } }
I20260619 08:19:09.739156 8321 raft_consensus.cc:388] T 00000000000000000000000000000000 P e81f7c56da5946ac996d06b7c1638c45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:19:09.739251 8321 raft_consensus.cc:482] T 00000000000000000000000000000000 P e81f7c56da5946ac996d06b7c1638c45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:19:09.739320 8321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e81f7c56da5946ac996d06b7c1638c45 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:09.740118 8321 raft_consensus.cc:504] T 00000000000000000000000000000000 P e81f7c56da5946ac996d06b7c1638c45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e81f7c56da5946ac996d06b7c1638c45" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43463 } }
I20260619 08:19:09.740218 8321 leader_election.cc:302] T 00000000000000000000000000000000 P e81f7c56da5946ac996d06b7c1638c45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e81f7c56da5946ac996d06b7c1638c45; no voters:
I20260619 08:19:09.740477 8321 leader_election.cc:288] T 00000000000000000000000000000000 P e81f7c56da5946ac996d06b7c1638c45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:19:09.740509 8326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e81f7c56da5946ac996d06b7c1638c45 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:09.740746 8326 raft_consensus.cc:686] T 00000000000000000000000000000000 P e81f7c56da5946ac996d06b7c1638c45 [term 1 LEADER]: Becoming Leader. State: Replica: e81f7c56da5946ac996d06b7c1638c45, State: Running, Role: LEADER
I20260619 08:19:09.740899 8321 sys_catalog.cc:564] T 00000000000000000000000000000000 P e81f7c56da5946ac996d06b7c1638c45 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:19:09.740878 8326 consensus_queue.cc:237] T 00000000000000000000000000000000 P e81f7c56da5946ac996d06b7c1638c45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e81f7c56da5946ac996d06b7c1638c45" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43463 } }
I20260619 08:19:09.741268 8326 sys_catalog.cc:455] T 00000000000000000000000000000000 P e81f7c56da5946ac996d06b7c1638c45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e81f7c56da5946ac996d06b7c1638c45. Latest consensus state: current_term: 1 leader_uuid: "e81f7c56da5946ac996d06b7c1638c45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e81f7c56da5946ac996d06b7c1638c45" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43463 } } }
I20260619 08:19:09.741345 8326 sys_catalog.cc:458] T 00000000000000000000000000000000 P e81f7c56da5946ac996d06b7c1638c45 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:09.741780 8327 sys_catalog.cc:455] T 00000000000000000000000000000000 P e81f7c56da5946ac996d06b7c1638c45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e81f7c56da5946ac996d06b7c1638c45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e81f7c56da5946ac996d06b7c1638c45" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43463 } } }
I20260619 08:19:09.741971 8327 sys_catalog.cc:458] T 00000000000000000000000000000000 P e81f7c56da5946ac996d06b7c1638c45 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:09.742551 8331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:19:09.743211 8331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:19:09.745038 8331 catalog_manager.cc:1347] Generated new cluster ID: 2ccdc6b83d754ce49fea92ea385e72b1
I20260619 08:19:09.745097 8331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:19:09 dist-test-slave-d3n8 krb5kdc[8240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:09.750916 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:09.738627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33108 (local address 127.4.151.190:43463)
0619 08:19:09.739275 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:09.739287 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:09.739324 (+ 37us) server_negotiation.cc:407] Connection header received
0619 08:19:09.739469 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:09.739478 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:09.739560 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:09.739730 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:09.740609 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:09.742237 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:09.743428 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:09.743989 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:09.744101 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:09.747111 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:09.747115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:09.747116 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:09.747130 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:09.749067 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:09.749558 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:09.749561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:09.749561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:09.749686 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:09.749943 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:09.749944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:09.749944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:09.750227 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:09.750416 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:09.750552 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 08:19:09.750709 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":531,"threads_started":1}
I20260619 08:19:09.759594 8331 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:19:09.760205 8331 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:19:09.772819 8331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e81f7c56da5946ac996d06b7c1638c45: Generated new TSK 0
I20260619 08:19:09.773180 8331 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.151.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
I20260619 08:19:09.810547 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43463
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:09.918220 8347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:09.918480 8347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:09.918520 8347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:09.919740 8347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:09.919817 8347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:09.919864 8347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:19:09.921509 8347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:09.921777 8347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:09.922084 8347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:09.930331 8353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:09.930541 8355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:09.930559 8352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:09.930800 8347 server_base.cc:1034] running on GCE node
I20260619 08:19:09.931030 8347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:09.931360 8347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:09.932524 8347 hybrid_clock.cc:648] HybridClock initialized: now 1781857149932458 us; error 75 us; skew 500 ppm
Jun 19 08:19:09 dist-test-slave-d3n8 krb5kdc[8240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857149, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:09.935580 8347 init.cc:377] Logged in from keytab as kudu/127.4.151.129@KRBTEST.COM (short username kudu)
I20260619 08:19:09.936203 8347 webserver.cc:460] Webserver started at http://127.4.151.129:45039/ using document root <none> and password file <none>
I20260619 08:19:09.936530 8347 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:09.936578 8347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:09.936673 8347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:09.938156 8347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "72cc479875d749c6a85f6167c5ca6be4"
format_stamp: "Formatted at 2026-06-19 08:19:09 on dist-test-slave-d3n8"
server_key: "e2ebfdc6e537c7c1b6fc5ac5bc47616c"
server_key_iv: "164cf350020051edfefab600d0e248c9"
server_key_version: "encryptionkey@0"
I20260619 08:19:09.938660 8347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "72cc479875d749c6a85f6167c5ca6be4"
format_stamp: "Formatted at 2026-06-19 08:19:09 on dist-test-slave-d3n8"
server_key: "e2ebfdc6e537c7c1b6fc5ac5bc47616c"
server_key_iv: "164cf350020051edfefab600d0e248c9"
server_key_version: "encryptionkey@0"
I20260619 08:19:09.940574 8347 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 08:19:09.941599 8362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:09.941757 8347 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:09.941854 8347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "72cc479875d749c6a85f6167c5ca6be4"
format_stamp: "Formatted at 2026-06-19 08:19:09 on dist-test-slave-d3n8"
server_key: "e2ebfdc6e537c7c1b6fc5ac5bc47616c"
server_key_iv: "164cf350020051edfefab600d0e248c9"
server_key_version: "encryptionkey@0"
I20260619 08:19:09.941944 8347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:09.961807 8347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:09.964565 8347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:09.964731 8347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:09.965010 8347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:09.965430 8347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:09.965478 8347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:09.965512 8347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:09.965548 8347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:09.973326 8347 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:32977
I20260619 08:19:09.973417 8475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:32977 every 8 connection(s)
I20260619 08:19:09.973814 8347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
I20260619 08:19:09.977174 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 8347
I20260619 08:19:09.977288 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:19:09.977514 4702 external_mini_cluster.cc:1428] Setting key c8c1d7eccf1dedeb9cd670ef966d4b46
Jun 19 08:19:09 dist-test-slave-d3n8 krb5kdc[8240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857149, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:09.985947 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:09.974796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:40855 (local address 127.4.151.190:43463)
0619 08:19:09.974851 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:09.974854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:09.975131 (+ 277us) server_negotiation.cc:407] Connection header received
0619 08:19:09.976316 (+ 1185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:09.976317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:09.976333 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:09.976394 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:09.977042 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:09.977816 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:09.979480 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:09.979711 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:09.979817 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:09.982683 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:09.982685 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:09.982685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:09.982693 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:09.984477 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:09.984895 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:09.984897 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:09.984897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:09.984961 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:09.985205 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:09.985206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:09.985206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:09.985358 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:09.985474 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:09.985749 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 08:19:09.985838 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 08:19:09.986030 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:09.974769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:43463 (local address 127.4.151.129:40855)
0619 08:19:09.974965 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:09.975022 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:09.976192 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:09.976450 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:09.976453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:09.976718 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:09.976912 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:09.976916 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:09.977916 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:09.977917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:09.979334 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:09.979338 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:09.979827 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:09.979829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:09.979960 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:09.980664 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:09.980673 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:09.982579 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:09.984572 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:09.984579 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:09.984582 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:09.984817 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:09.985027 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:09.985028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:09.985028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:09.985139 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:09.985467 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:09.985473 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:09.985611 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 08:19:09.985703 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 08:19:09.985832 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":88,"threads_started":1}
I20260619 08:19:09.986670 8476 heartbeater.cc:347] Connected to a master server at 127.4.151.190:43463
I20260619 08:19:09.986760 8476 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:09.986965 8476 heartbeater.cc:511] Master 127.4.151.190:43463 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.151.130@KRBTEST.COM; defaulting to no policy
I20260619 08:19:09.987527 8285 ts_manager.cc:194] Registered new tserver with Master: 72cc479875d749c6a85f6167c5ca6be4 (127.4.151.129:32977)
I20260619 08:19:09.988188 8285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.129@KRBTEST.COM'} at 127.4.151.129:40855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
I20260619 08:19:10.009440 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43463
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:10.093720 8481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:10.094127 8481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:10.094164 8481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:10.095397 8481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:10.095438 8481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:10.095465 8481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:19:10.096993 8481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:10.097187 8481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:10.097368 8481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:10.102692 8486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:10.102682 8489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:10.102681 8487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:10.103046 8481 server_base.cc:1034] running on GCE node
I20260619 08:19:10.103299 8481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:10.103551 8481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:10.104772 8481 hybrid_clock.cc:648] HybridClock initialized: now 1781857150104767 us; error 39 us; skew 500 ppm
Jun 19 08:19:10 dist-test-slave-d3n8 krb5kdc[8240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857150, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:10.107918 8481 init.cc:377] Logged in from keytab as kudu/127.4.151.130@KRBTEST.COM (short username kudu)
I20260619 08:19:10.108384 8481 webserver.cc:460] Webserver started at http://127.4.151.130:39495/ using document root <none> and password file <none>
I20260619 08:19:10.108651 8481 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:10.108727 8481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:10.108870 8481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:10.110042 8481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "59b06e57eda54c83b03cc7c15f5f37ff"
format_stamp: "Formatted at 2026-06-19 08:19:10 on dist-test-slave-d3n8"
server_key: "55eb7691d6264ffeda617a507d3792d1"
server_key_iv: "9ff4bfa5a953b722da8ed5090acfc99b"
server_key_version: "encryptionkey@0"
I20260619 08:19:10.110430 8481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "59b06e57eda54c83b03cc7c15f5f37ff"
format_stamp: "Formatted at 2026-06-19 08:19:10 on dist-test-slave-d3n8"
server_key: "55eb7691d6264ffeda617a507d3792d1"
server_key_iv: "9ff4bfa5a953b722da8ed5090acfc99b"
server_key_version: "encryptionkey@0"
I20260619 08:19:10.112063 8481 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 08:19:10.112907 8496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:10.113238 8481 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:10.113355 8481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "59b06e57eda54c83b03cc7c15f5f37ff"
format_stamp: "Formatted at 2026-06-19 08:19:10 on dist-test-slave-d3n8"
server_key: "55eb7691d6264ffeda617a507d3792d1"
server_key_iv: "9ff4bfa5a953b722da8ed5090acfc99b"
server_key_version: "encryptionkey@0"
I20260619 08:19:10.113518 8481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:10.120524 8481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:10.122884 8481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:10.123052 8481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:10.123278 8481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:10.123646 8481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:10.123704 8481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:10.123745 8481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:10.123773 8481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:10.129878 8481 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:38209
I20260619 08:19:10.129921 8609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:38209 every 8 connection(s)
I20260619 08:19:10.130288 8481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
I20260619 08:19:10.134711 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 8481
I20260619 08:19:10.134809 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:19:10.135111 4702 external_mini_cluster.cc:1428] Setting key 7fc15cbbfc0c65d4f04b507a571db8fb
Jun 19 08:19:10 dist-test-slave-d3n8 krb5kdc[8240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857150, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:10.141661 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:10.131050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:51021 (local address 127.4.151.190:43463)
0619 08:19:10.131113 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:10.131115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:10.131315 (+ 200us) server_negotiation.cc:407] Connection header received
0619 08:19:10.132281 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:10.132282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:10.132293 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:10.132365 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:10.133029 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:10.133675 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:10.134894 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:10.135197 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:10.135286 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:10.138000 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:10.138001 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:10.138002 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:10.138011 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:10.139868 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:10.140421 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:10.140422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:10.140423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:10.140493 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:10.140769 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:10.140770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:10.140771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:10.140958 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:10.141042 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:10.141398 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 08:19:10.141519 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 08:19:10.141762 8612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:10.131029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:43463 (local address 127.4.151.130:51021)
0619 08:19:10.131175 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:10.131225 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:10.132158 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:10.132428 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:10.132432 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:10.132690 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:10.132894 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:10.132898 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:10.133787 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:10.133788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:10.134740 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:10.134743 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:10.135415 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:10.135416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:10.135539 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:10.136138 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:10.136146 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:10.137844 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:10.139982 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:10.139992 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:10.139994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:10.140324 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:10.140573 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:10.140573 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:10.140574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:10.140695 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:10.141088 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:10.141096 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:10.141232 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 08:19:10.141347 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 08:19:10.141564 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260619 08:19:10.142336 8610 heartbeater.cc:347] Connected to a master server at 127.4.151.190:43463
I20260619 08:19:10.142457 8610 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:10.142643 8610 heartbeater.cc:511] Master 127.4.151.190:43463 requested a full tablet report, sending...
I20260619 08:19:10.143153 8285 ts_manager.cc:194] Registered new tserver with Master: 59b06e57eda54c83b03cc7c15f5f37ff (127.4.151.130:38209)
I20260619 08:19:10.143709 8285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.130@KRBTEST.COM'} at 127.4.151.130:51021
WARNING: no policy specified for kudu/127.4.151.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
I20260619 08:19:10.165647 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43463
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:10.248867 8615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:10.249106 8615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:10.249133 8615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:10.250535 8615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:10.250589 8615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:10.250618 8615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:19:10.252184 8615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:10.252390 8615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:10.252586 8615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:10.257722 8621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:10.257727 8620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:10.257719 8623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:10.257918 8615 server_base.cc:1034] running on GCE node
I20260619 08:19:10.258198 8615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:10.258402 8615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:10.259570 8615 hybrid_clock.cc:648] HybridClock initialized: now 1781857150259506 us; error 81 us; skew 500 ppm
Jun 19 08:19:10 dist-test-slave-d3n8 krb5kdc[8240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857150, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:10.262354 8615 init.cc:377] Logged in from keytab as kudu/127.4.151.131@KRBTEST.COM (short username kudu)
I20260619 08:19:10.262857 8615 webserver.cc:460] Webserver started at http://127.4.151.131:46881/ using document root <none> and password file <none>
I20260619 08:19:10.263094 8615 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:10.263160 8615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:10.263321 8615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:10.264405 8615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "e3bc77d8368f4262afa3132083b69352"
format_stamp: "Formatted at 2026-06-19 08:19:10 on dist-test-slave-d3n8"
server_key: "7d435973957a8ff50185b7ae0b349b69"
server_key_iv: "9939581fef7fbaab9f6c7d645d786e93"
server_key_version: "encryptionkey@0"
I20260619 08:19:10.264780 8615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "e3bc77d8368f4262afa3132083b69352"
format_stamp: "Formatted at 2026-06-19 08:19:10 on dist-test-slave-d3n8"
server_key: "7d435973957a8ff50185b7ae0b349b69"
server_key_iv: "9939581fef7fbaab9f6c7d645d786e93"
server_key_version: "encryptionkey@0"
I20260619 08:19:10.266356 8615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 08:19:10.267387 8630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:10.267570 8615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:10.267663 8615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "e3bc77d8368f4262afa3132083b69352"
format_stamp: "Formatted at 2026-06-19 08:19:10 on dist-test-slave-d3n8"
server_key: "7d435973957a8ff50185b7ae0b349b69"
server_key_iv: "9939581fef7fbaab9f6c7d645d786e93"
server_key_version: "encryptionkey@0"
I20260619 08:19:10.267728 8615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:10.275765 8615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:10.277935 8615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:10.278100 8615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:10.278317 8615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:10.278613 8615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:10.278666 8615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:10.278709 8615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:10.278738 8615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:10.284685 8615 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:43503
I20260619 08:19:10.284726 8743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:43503 every 8 connection(s)
I20260619 08:19:10.285079 8615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
I20260619 08:19:10.290735 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 8615
I20260619 08:19:10.290810 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:19:10.291007 4702 external_mini_cluster.cc:1428] Setting key 57697359bf50a5df2baf9d84211eb143
Jun 19 08:19:10 dist-test-slave-d3n8 krb5kdc[8240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857150, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:10.295678 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:10.285974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:49385 (local address 127.4.151.190:43463)
0619 08:19:10.286082 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:10.286085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:10.286272 (+ 187us) server_negotiation.cc:407] Connection header received
0619 08:19:10.287382 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:10.287384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:10.287397 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:10.287457 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:10.288102 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:10.288619 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:10.289689 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:10.289893 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:10.289984 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:10.292632 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:10.292633 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:10.292634 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:10.292642 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:10.294194 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:10.294667 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:10.294668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:10.294669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:10.294730 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:10.295011 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:10.295012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:10.295012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:10.295134 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:10.295216 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:10.295498 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 08:19:10.295576 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 08:19:10.295872 8746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:10.285997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:43463 (local address 127.4.151.131:49385)
0619 08:19:10.286148 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:10.286184 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:10.287244 (+ 1060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:10.287532 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:10.287535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:10.287788 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:10.287992 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:10.287996 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:10.288726 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:10.288727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:10.289599 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:10.289601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:10.289994 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:10.289995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:10.290142 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:10.290714 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:10.290722 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:10.292529 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:10.294316 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:10.294322 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:10.294325 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:10.294565 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:10.294799 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:10.294800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:10.294801 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:10.294939 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:10.295202 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:10.295208 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:10.295354 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 08:19:10.295477 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 08:19:10.295637 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260619 08:19:10.296344 8744 heartbeater.cc:347] Connected to a master server at 127.4.151.190:43463
I20260619 08:19:10.296463 8744 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:10.296600 8744 heartbeater.cc:511] Master 127.4.151.190:43463 requested a full tablet report, sending...
I20260619 08:19:10.296957 8285 ts_manager.cc:194] Registered new tserver with Master: e3bc77d8368f4262afa3132083b69352 (127.4.151.131:43503)
I20260619 08:19:10.297329 8285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.131@KRBTEST.COM'} at 127.4.151.131:49385
I20260619 08:19:10.302853 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 08:19:10 dist-test-slave-d3n8 krb5kdc[8240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 08:19:10 dist-test-slave-d3n8 krb5kdc[8240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.129@KRBTEST.COM
I20260619 08:19:10.326785 8751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:10.318425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42080 (local address 127.4.151.129:32977)
0619 08:19:10.318601 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:10.318606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:10.318617 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:19:10.318657 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:10.318659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:10.318673 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:10.318766 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:10.319509 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:10.320286 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:10.321097 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:10.321501 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:10.321601 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:10.323836 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:10.323845 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:10.323845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:10.323853 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:10.325436 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:10.325805 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:10.325808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:10.325808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:10.325881 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:10.326156 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:10.326156 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:10.326157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:10.326305 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:10.326466 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:10.326570 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 08:19:10.326650 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
Jun 19 08:19:10 dist-test-slave-d3n8 krb5kdc[8240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:10.335927 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:10.328201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33118 (local address 127.4.151.190:43463)
0619 08:19:10.328271 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:10.328273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:10.328282 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:19:10.328343 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:10.328344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:10.328354 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:10.328409 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:10.328925 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:10.329495 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:10.330330 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:10.330536 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:10.330621 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:10.332844 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:10.332846 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:10.332846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:10.332854 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:10.334578 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:10.334988 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:10.334989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:10.334990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:10.335052 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:10.335286 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:10.335287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:10.335288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:10.335447 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:10.335556 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:10.335678 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 08:19:10.335760 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 08:19:10.336212 8285 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33118
I20260619 08:19:10.336621 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 8347
I20260619 08:19:10.342308 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 8481
I20260619 08:19:10.347100 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 8615
I20260619 08:19:10.351560 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 8253
2026-06-19T08:19:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1935 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:19:10 dist-test-slave-d3n8 krb5kdc[8756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:19:10 dist-test-slave-d3n8 krb5kdc[8756](info): set up 2 sockets
Jun 19 08:19:10 dist-test-slave-d3n8 krb5kdc[8756](info): commencing operation
krb5kdc: starting...
W20260619 08:19:11.389536 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:19:11 dist-test-slave-d3n8 krb5kdc[8756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:19:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:19:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:19:11.517810 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:43591
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:40359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:43591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:11.601599 8769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:11.601862 8769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:11.601902 8769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:11.603154 8769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:19:11.603226 8769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:11.603250 8769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:11.603271 8769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:19:11.603302 8769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:19:11.604853 8769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:40359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:43591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:43591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:11.605134 8769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:11.605393 8769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:11.610494 8777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:11.610581 8775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:11.610533 8774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:11.610704 8769 server_base.cc:1034] running on GCE node
I20260619 08:19:11.610931 8769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:11.611239 8769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:11.612402 8769 hybrid_clock.cc:648] HybridClock initialized: now 1781857151612379 us; error 35 us; skew 500 ppm
Jun 19 08:19:11 dist-test-slave-d3n8 krb5kdc[8756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857151, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:11.618342 8769 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:19:11.618696 8769 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:11.618770 8769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:11.618920 8769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:11.620041 8769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "b103b400de644931826acced764cec3b"
format_stamp: "Formatted at 2026-06-19 08:19:11 on dist-test-slave-d3n8"
server_key: "750d9f95f88d6e7432bf77f5c22697c9"
server_key_iv: "6978dbd52202d334b25e8e6a605923cf"
server_key_version: "encryptionkey@0"
I20260619 08:19:11.620445 8769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "b103b400de644931826acced764cec3b"
format_stamp: "Formatted at 2026-06-19 08:19:11 on dist-test-slave-d3n8"
server_key: "750d9f95f88d6e7432bf77f5c22697c9"
server_key_iv: "6978dbd52202d334b25e8e6a605923cf"
server_key_version: "encryptionkey@0"
I20260619 08:19:11.621860 8769 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:19:11.622673 8783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:11.622886 8769 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:11.623028 8769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "b103b400de644931826acced764cec3b"
format_stamp: "Formatted at 2026-06-19 08:19:11 on dist-test-slave-d3n8"
server_key: "750d9f95f88d6e7432bf77f5c22697c9"
server_key_iv: "6978dbd52202d334b25e8e6a605923cf"
server_key_version: "encryptionkey@0"
I20260619 08:19:11.623111 8769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:11.631656 8769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:11.633895 8769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:11.634037 8769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:11.638114 8769 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:43591
I20260619 08:19:11.638206 8835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:43591 every 8 connection(s)
I20260619 08:19:11.638474 8769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:19:11.639555 8836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:11.642257 8836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b103b400de644931826acced764cec3b: Bootstrap starting.
I20260619 08:19:11.642760 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 8769
I20260619 08:19:11.642894 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:19:11.643085 8836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b103b400de644931826acced764cec3b: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:11.643119 4702 external_mini_cluster.cc:1428] Setting key 5f27b5bfd2a7445e18955ddfe80cbde3
I20260619 08:19:11.643440 8836 log.cc:824] T 00000000000000000000000000000000 P b103b400de644931826acced764cec3b: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:11.644234 8836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b103b400de644931826acced764cec3b: No bootstrap required, opened a new log
I20260619 08:19:11.646332 8836 raft_consensus.cc:348] T 00000000000000000000000000000000 P b103b400de644931826acced764cec3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b103b400de644931826acced764cec3b" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43591 } }
I20260619 08:19:11.646481 8836 raft_consensus.cc:374] T 00000000000000000000000000000000 P b103b400de644931826acced764cec3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:11.646507 8836 raft_consensus.cc:729] T 00000000000000000000000000000000 P b103b400de644931826acced764cec3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b103b400de644931826acced764cec3b, State: Initialized, Role: FOLLOWER
I20260619 08:19:11.646656 8836 consensus_queue.cc:260] T 00000000000000000000000000000000 P b103b400de644931826acced764cec3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b103b400de644931826acced764cec3b" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43591 } }
I20260619 08:19:11.646745 8836 raft_consensus.cc:388] T 00000000000000000000000000000000 P b103b400de644931826acced764cec3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:19:11.646792 8836 raft_consensus.cc:482] T 00000000000000000000000000000000 P b103b400de644931826acced764cec3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:19:11.646844 8836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b103b400de644931826acced764cec3b [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:11.647449 8836 raft_consensus.cc:504] T 00000000000000000000000000000000 P b103b400de644931826acced764cec3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b103b400de644931826acced764cec3b" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43591 } }
I20260619 08:19:11.647579 8836 leader_election.cc:302] T 00000000000000000000000000000000 P b103b400de644931826acced764cec3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b103b400de644931826acced764cec3b; no voters:
I20260619 08:19:11.647792 8836 leader_election.cc:288] T 00000000000000000000000000000000 P b103b400de644931826acced764cec3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:19:11.647822 8841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b103b400de644931826acced764cec3b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:11.648028 8841 raft_consensus.cc:686] T 00000000000000000000000000000000 P b103b400de644931826acced764cec3b [term 1 LEADER]: Becoming Leader. State: Replica: b103b400de644931826acced764cec3b, State: Running, Role: LEADER
I20260619 08:19:11.648164 8841 consensus_queue.cc:237] T 00000000000000000000000000000000 P b103b400de644931826acced764cec3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b103b400de644931826acced764cec3b" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43591 } }
I20260619 08:19:11.648279 8836 sys_catalog.cc:564] T 00000000000000000000000000000000 P b103b400de644931826acced764cec3b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:19:11.648545 8841 sys_catalog.cc:455] T 00000000000000000000000000000000 P b103b400de644931826acced764cec3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b103b400de644931826acced764cec3b. Latest consensus state: current_term: 1 leader_uuid: "b103b400de644931826acced764cec3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b103b400de644931826acced764cec3b" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43591 } } }
I20260619 08:19:11.648620 8841 sys_catalog.cc:458] T 00000000000000000000000000000000 P b103b400de644931826acced764cec3b [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:11.648605 8843 sys_catalog.cc:455] T 00000000000000000000000000000000 P b103b400de644931826acced764cec3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b103b400de644931826acced764cec3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b103b400de644931826acced764cec3b" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43591 } } }
I20260619 08:19:11.648703 8843 sys_catalog.cc:458] T 00000000000000000000000000000000 P b103b400de644931826acced764cec3b [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:11.648981 8847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:19:11.649578 8847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 08:19:11 dist-test-slave-d3n8 krb5kdc[8756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:11.651665 8847 catalog_manager.cc:1347] Generated new cluster ID: 6e05aef1699c46bcbc11df5e1ec9ff60
I20260619 08:19:11.651749 8847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:19:11.654964 8840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:11.643686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47904 (local address 127.4.151.190:43591)
0619 08:19:11.643857 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:11.643867 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:11.643903 (+ 36us) server_negotiation.cc:407] Connection header received
0619 08:19:11.644442 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:11.644449 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:11.644523 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:11.644674 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:11.645320 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:11.646246 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:11.647256 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:11.647754 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:11.647870 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:11.651060 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:11.651064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:11.651064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:11.651076 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:11.653179 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:11.653636 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:11.653639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:11.653639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:11.653746 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:11.653980 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:11.653981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:11.653981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:11.654301 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:11.654476 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:11.654586 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 08:19:11.654730 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260619 08:19:11.662663 8847 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:19:11.663301 8847 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:19:11.672547 8847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b103b400de644931826acced764cec3b: Generated new TSK 0
I20260619 08:19:11.672780 8847 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.151.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
I20260619 08:19:11.709739 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43591
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:40359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:11.792236 8862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:11.792459 8862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:11.792485 8862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:11.793725 8862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:11.793772 8862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:11.793802 8862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:19:11.795396 8862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:40359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:11.795660 8862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:11.795885 8862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:11.800969 8867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:11.800971 8870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:11.801009 8868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:11.801331 8862 server_base.cc:1034] running on GCE node
I20260619 08:19:11.801610 8862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:11.801855 8862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:11.803071 8862 hybrid_clock.cc:648] HybridClock initialized: now 1781857151803047 us; error 36 us; skew 500 ppm
Jun 19 08:19:11 dist-test-slave-d3n8 krb5kdc[8756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857151, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:11.806011 8862 init.cc:377] Logged in from keytab as kudu/127.4.151.129@KRBTEST.COM (short username kudu)
I20260619 08:19:11.806428 8862 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:11.806566 8862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:11.806721 8862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:11.807818 8862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "03d2d4fa8a0d429e82d7a36943172f08"
format_stamp: "Formatted at 2026-06-19 08:19:11 on dist-test-slave-d3n8"
server_key: "c0a9a61a51038a1ec992c2d0062146f3"
server_key_iv: "aae02b544a9a9ec8ae42d947c209fb37"
server_key_version: "encryptionkey@0"
I20260619 08:19:11.808200 8862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "03d2d4fa8a0d429e82d7a36943172f08"
format_stamp: "Formatted at 2026-06-19 08:19:11 on dist-test-slave-d3n8"
server_key: "c0a9a61a51038a1ec992c2d0062146f3"
server_key_iv: "aae02b544a9a9ec8ae42d947c209fb37"
server_key_version: "encryptionkey@0"
I20260619 08:19:11.809669 8862 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:19:11.810529 8876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:11.810709 8862 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:11.810814 8862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "03d2d4fa8a0d429e82d7a36943172f08"
format_stamp: "Formatted at 2026-06-19 08:19:11 on dist-test-slave-d3n8"
server_key: "c0a9a61a51038a1ec992c2d0062146f3"
server_key_iv: "aae02b544a9a9ec8ae42d947c209fb37"
server_key_version: "encryptionkey@0"
I20260619 08:19:11.810891 8862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:11.826753 8862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:11.829298 8862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:11.829522 8862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:11.829774 8862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:11.830109 8862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:11.830166 8862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:11.830207 8862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:11.830240 8862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:11.836689 8862 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:41031
I20260619 08:19:11.836725 8989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:41031 every 8 connection(s)
I20260619 08:19:11.837016 8862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
Jun 19 08:19:11 dist-test-slave-d3n8 krb5kdc[8756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857151, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:11.844995 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 8862
I20260619 08:19:11.845142 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:19:11.845350 4702 external_mini_cluster.cc:1428] Setting key ea838c307b29a034e3b8e8fa2c0b6cd9
I20260619 08:19:11.847411 8840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:11.837852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:55081 (local address 127.4.151.190:43591)
0619 08:19:11.837943 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:11.837945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:11.838192 (+ 247us) server_negotiation.cc:407] Connection header received
0619 08:19:11.838832 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:11.838833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:11.838845 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:11.838909 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:11.840054 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:11.840579 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:11.841594 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:11.841801 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:11.841864 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:11.844206 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:11.844207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:11.844207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:11.844214 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:11.845899 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:11.846388 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:11.846388 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:11.846389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:11.846444 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:11.846767 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:11.846768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:11.846768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:11.846879 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:11.846929 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:11.847270 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 08:19:11.847345 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 08:19:11.847927 8992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:11.837809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:43591 (local address 127.4.151.129:55081)
0619 08:19:11.838054 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:11.838097 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:11.838594 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:11.839175 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:11.839179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:11.839448 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:11.839923 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:11.839927 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:11.840662 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:11.840663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:11.841516 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:11.841518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:11.841881 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:11.841882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:11.842016 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:11.842532 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:11.842541 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:11.844104 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:11.845963 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:11.845971 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:11.845973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:11.846306 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:11.846513 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:11.846514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:11.846514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:11.846706 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:11.846942 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:11.846947 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:11.847129 (+ 182us) client_negotiation.cc:769] Sending connection context
0619 08:19:11.847249 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 08:19:11.847436 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":88,"threads_started":1}
I20260619 08:19:11.848408 8990 heartbeater.cc:347] Connected to a master server at 127.4.151.190:43591
I20260619 08:19:11.848531 8990 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:11.848690 8990 heartbeater.cc:511] Master 127.4.151.190:43591 requested a full tablet report, sending...
I20260619 08:19:11.849299 8800 ts_manager.cc:194] Registered new tserver with Master: 03d2d4fa8a0d429e82d7a36943172f08 (127.4.151.129:41031)
I20260619 08:19:11.850283 8800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.129@KRBTEST.COM'} at 127.4.151.129:55081
WARNING: no policy specified for kudu/127.4.151.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
I20260619 08:19:11.876757 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43591
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:40359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:11.959270 8995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:11.959491 8995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:11.959514 8995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:11.960697 8995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:11.960744 8995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:11.960772 8995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:19:11.962445 8995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:40359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:11.962723 8995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:11.962972 8995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:11.968628 9001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:11.968628 9000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:11.968624 9003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:11.968794 8995 server_base.cc:1034] running on GCE node
I20260619 08:19:11.969184 8995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:11.969501 8995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:11.970669 8995 hybrid_clock.cc:648] HybridClock initialized: now 1781857151970637 us; error 48 us; skew 500 ppm
Jun 19 08:19:11 dist-test-slave-d3n8 krb5kdc[8756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857151, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:11.973649 8995 init.cc:377] Logged in from keytab as kudu/127.4.151.130@KRBTEST.COM (short username kudu)
I20260619 08:19:11.974086 8995 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:11.974140 8995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:11.974298 8995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:11.975389 8995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "33f33cf4204a4962b6714bc0487b7bf0"
format_stamp: "Formatted at 2026-06-19 08:19:11 on dist-test-slave-d3n8"
server_key: "db716cc292ce9df5fc2edf1de8ec6fdc"
server_key_iv: "f41572922f77774c4abefcd79f4678b5"
server_key_version: "encryptionkey@0"
I20260619 08:19:11.975739 8995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "33f33cf4204a4962b6714bc0487b7bf0"
format_stamp: "Formatted at 2026-06-19 08:19:11 on dist-test-slave-d3n8"
server_key: "db716cc292ce9df5fc2edf1de8ec6fdc"
server_key_iv: "f41572922f77774c4abefcd79f4678b5"
server_key_version: "encryptionkey@0"
I20260619 08:19:11.977180 8995 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:19:11.978132 9009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:11.978343 8995 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:11.978441 8995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "33f33cf4204a4962b6714bc0487b7bf0"
format_stamp: "Formatted at 2026-06-19 08:19:11 on dist-test-slave-d3n8"
server_key: "db716cc292ce9df5fc2edf1de8ec6fdc"
server_key_iv: "f41572922f77774c4abefcd79f4678b5"
server_key_version: "encryptionkey@0"
I20260619 08:19:11.978520 8995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:11.991039 8995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:11.993453 8995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:11.993599 8995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:11.993863 8995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:11.994194 8995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:11.994227 8995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:11.994249 8995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:11.994297 8995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:11.999948 8995 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:46517
I20260619 08:19:12.000103 9122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:46517 every 8 connection(s)
I20260619 08:19:12.000304 8995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
I20260619 08:19:12.002345 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 8995
I20260619 08:19:12.002486 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:19:12.002705 4702 external_mini_cluster.cc:1428] Setting key f15b46e8b8e4b7dfd604f537c2c645f6
Jun 19 08:19:12 dist-test-slave-d3n8 krb5kdc[8756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857151, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.151.131@KRBTEST.COM; defaulting to no policy
I20260619 08:19:12.012562 8840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.001005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:38049 (local address 127.4.151.190:43591)
0619 08:19:12.001098 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:12.001100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:12.001311 (+ 211us) server_negotiation.cc:407] Connection header received
0619 08:19:12.001883 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:12.001884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:12.001970 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:12.002209 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:12.003432 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.004189 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.005728 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.005923 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.006076 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:12.009136 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:12.009138 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:12.009139 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:12.009150 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:12.011052 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:12.011502 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:12.011504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:12.011504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:12.011573 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:12.011820 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:12.011822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:12.011822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:12.011988 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:12.012082 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:12.012345 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 08:19:12.012438 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 08:19:12.012713 9125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.000988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:43591 (local address 127.4.151.130:38049)
0619 08:19:12.001174 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:12.001216 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:12.001764 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:12.002284 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:12.002288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:12.002683 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:12.003315 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:12.003321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.004290 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.004291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:12.005610 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:12.005614 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.006336 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.006339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:12.006516 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:12.007249 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:12.007260 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:12.009016 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:12.011157 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:12.011165 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:12.011167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:12.011409 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:12.011648 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:12.011649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:12.011649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:12.011750 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:12.012070 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:12.012075 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:12.012200 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 08:19:12.012291 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 08:19:12.012471 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":112,"threads_started":1}
I20260619 08:19:12.013271 9123 heartbeater.cc:347] Connected to a master server at 127.4.151.190:43591
I20260619 08:19:12.013382 9123 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:12.013582 9123 heartbeater.cc:511] Master 127.4.151.190:43591 requested a full tablet report, sending...
I20260619 08:19:12.014024 8800 ts_manager.cc:194] Registered new tserver with Master: 33f33cf4204a4962b6714bc0487b7bf0 (127.4.151.130:46517)
I20260619 08:19:12.014400 8800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.130@KRBTEST.COM'} at 127.4.151.130:38049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
I20260619 08:19:12.032331 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43591
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:40359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:12.114195 9128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:12.114452 9128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:12.114480 9128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:12.115608 9128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:12.115644 9128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:12.115670 9128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:19:12.117126 9128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:40359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:12.117322 9128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:12.117597 9128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:12.122794 9136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:12.122797 9133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:12.122830 9128 server_base.cc:1034] running on GCE node
W20260619 08:19:12.122797 9134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:12.123369 9128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:12.123629 9128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:12.124818 9128 hybrid_clock.cc:648] HybridClock initialized: now 1781857152124818 us; error 41 us; skew 500 ppm
Jun 19 08:19:12 dist-test-slave-d3n8 krb5kdc[8756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857152, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:12.127630 9128 init.cc:377] Logged in from keytab as kudu/127.4.151.131@KRBTEST.COM (short username kudu)
I20260619 08:19:12.127974 9128 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:12.128023 9128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:12.128126 9128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:12.129150 9128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "599243d5b3144521911affeebc99c92b"
format_stamp: "Formatted at 2026-06-19 08:19:12 on dist-test-slave-d3n8"
server_key: "cc98584d4ca9ef570b2a474c5981514e"
server_key_iv: "333d3d92e467f33c5380195cf24b2804"
server_key_version: "encryptionkey@0"
I20260619 08:19:12.129535 9128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "599243d5b3144521911affeebc99c92b"
format_stamp: "Formatted at 2026-06-19 08:19:12 on dist-test-slave-d3n8"
server_key: "cc98584d4ca9ef570b2a474c5981514e"
server_key_iv: "333d3d92e467f33c5380195cf24b2804"
server_key_version: "encryptionkey@0"
I20260619 08:19:12.130806 9128 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:12.131582 9142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:12.131780 9128 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:12.131875 9128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "599243d5b3144521911affeebc99c92b"
format_stamp: "Formatted at 2026-06-19 08:19:12 on dist-test-slave-d3n8"
server_key: "cc98584d4ca9ef570b2a474c5981514e"
server_key_iv: "333d3d92e467f33c5380195cf24b2804"
server_key_version: "encryptionkey@0"
I20260619 08:19:12.131932 9128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:12.146284 9128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:12.148717 9128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:12.148836 9128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:12.149180 9128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:12.149595 9128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:12.149631 9128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:12.149655 9128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:12.149701 9128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:12.155726 9128 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:40319
I20260619 08:19:12.155778 9255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:40319 every 8 connection(s)
I20260619 08:19:12.156065 9128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
I20260619 08:19:12.157625 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 9128
I20260619 08:19:12.157749 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:19:12.157931 4702 external_mini_cluster.cc:1428] Setting key e6b272676683c57d21006d6673ab7b64
Jun 19 08:19:12 dist-test-slave-d3n8 krb5kdc[8756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857152, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:12.165491 8840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.156777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:51263 (local address 127.4.151.190:43591)
0619 08:19:12.156837 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:12.156840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:12.157002 (+ 162us) server_negotiation.cc:407] Connection header received
0619 08:19:12.157645 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:12.157647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:12.157660 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:12.157719 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:12.158787 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.159317 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.160346 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.160545 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.160641 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:12.162745 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:12.162746 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:12.162746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:12.162754 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:12.164061 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:12.164476 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:12.164477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:12.164478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:12.164538 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:12.164788 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:12.164789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:12.164790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:12.164907 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:12.164982 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:12.165287 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 08:19:12.165367 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 08:19:12.165679 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.156736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:43591 (local address 127.4.151.131:51263)
0619 08:19:12.156881 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:12.156918 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:12.157535 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:12.157792 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:12.157796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:12.158182 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:12.158684 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:12.158689 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.159416 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.159417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:12.160256 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:12.160258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.160647 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.160648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:12.160773 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:12.161281 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:12.161290 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:12.162653 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:12.164165 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:12.164172 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:12.164174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:12.164389 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:12.164619 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:12.164620 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:12.164620 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:12.164722 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:12.164974 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:12.164979 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:12.165155 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 08:19:12.165266 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 08:19:12.165445 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260619 08:19:12.166208 9256 heartbeater.cc:347] Connected to a master server at 127.4.151.190:43591
I20260619 08:19:12.166288 9256 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:12.166450 9256 heartbeater.cc:511] Master 127.4.151.190:43591 requested a full tablet report, sending...
I20260619 08:19:12.166842 8800 ts_manager.cc:194] Registered new tserver with Master: 599243d5b3144521911affeebc99c92b (127.4.151.131:40319)
I20260619 08:19:12.167254 8800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.131@KRBTEST.COM'} at 127.4.151.131:51263
I20260619 08:19:12.169559 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:19:12.176810 8840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.170832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47906 (local address 127.4.151.190:43591)
0619 08:19:12.170897 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:12.170900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:12.171005 (+ 105us) server_negotiation.cc:407] Connection header received
0619 08:19:12.171147 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:12.171148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:12.171158 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:12.171230 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:12.171787 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.172261 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.172976 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.173169 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.173240 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:12.174503 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:12.174505 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:12.174505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:12.174512 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:12.175671 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:12.176010 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:12.176011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:12.176011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:12.176073 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:12.176267 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:12.176268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:12.176269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:12.176419 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:12.176531 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:12.176630 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 08:19:12.176709 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 08:19:12.178537 8800 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:19:12.180102 8800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:19:12.186903 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.183272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51218 (local address 127.4.151.129:41031)
0619 08:19:12.183437 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:12.183443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:12.183455 (+ 12us) server_negotiation.cc:407] Connection header received
0619 08:19:12.183526 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:12.183528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:12.183547 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:12.183627 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:12.183895 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.184612 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.185827 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.186537 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.186613 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:12.186676 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:12.186808 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 08:19:12.186838 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":106,"threads_started":1}
I20260619 08:19:12.187008 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.183018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:41031 (local address 127.0.0.1:51218)
0619 08:19:12.183412 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:12.183421 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:12.183458 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:12.183734 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:12.183737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:12.183740 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:12.183812 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:12.183815 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.184708 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.184709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:12.185732 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:12.185735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.186611 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.186612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:12.186708 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:12.186721 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 08:19:12.186799 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 08:19:12.186848 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
I20260619 08:19:12.187609 9267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.182912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:40319 (local address 127.0.0.1:39206)
0619 08:19:12.183086 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:12.183125 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:12.183207 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:12.183767 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:12.183768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:12.183771 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:12.183828 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:12.183830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.185404 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.185406 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:12.186296 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:12.186298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.187347 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.187348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:12.187412 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:12.187416 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:19:12.187490 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 08:19:12.187531 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":1,"thread_start_us":64,"threads_started":1}
I20260619 08:19:12.187620 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.182989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39206 (local address 127.4.151.131:40319)
0619 08:19:12.183571 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:12.183575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:12.183582 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:19:12.183641 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:12.183643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:12.183659 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:12.183972 (+ 313us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:12.184048 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.185264 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.186393 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.187099 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.187187 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:12.187267 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:12.187506 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 08:19:12.187543 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":81,"threads_started":1}
I20260619 08:19:12.187796 8924 tablet_service.cc:1467] Processing CreateTablet for tablet 0f467a854f0e49c4985bedcc0cb93479 (DEFAULT_TABLE table=test-table [id=be1949e040144be48e13d94a614f4c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:12.188230 8924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f467a854f0e49c4985bedcc0cb93479. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:12.188361 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.183494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37804 (local address 127.4.151.130:46517)
0619 08:19:12.183671 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:12.183675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:12.183686 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:19:12.183720 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:12.183722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:12.183738 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:12.183939 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:12.184020 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.185236 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.187242 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.187976 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.188080 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:12.188133 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:12.188238 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 08:19:12.188284 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
I20260619 08:19:12.188638 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 0f467a854f0e49c4985bedcc0cb93479 (DEFAULT_TABLE table=test-table [id=be1949e040144be48e13d94a614f4c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:12.188931 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f467a854f0e49c4985bedcc0cb93479. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:12.189539 9269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.183018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:46517 (local address 127.0.0.1:37804)
0619 08:19:12.183249 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:12.183258 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:12.183313 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:12.183832 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:12.183833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:12.183834 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:12.183870 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:12.183871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.185362 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.185363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:12.186519 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:12.186523 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.188089 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.188090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:12.188156 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:12.188159 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:19:12.188223 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 08:19:12.189469 (+ 1246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":50,"thread_start_us":87,"threads_started":1}
I20260619 08:19:12.190300 9057 tablet_service.cc:1467] Processing CreateTablet for tablet 0f467a854f0e49c4985bedcc0cb93479 (DEFAULT_TABLE table=test-table [id=be1949e040144be48e13d94a614f4c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:12.190601 9057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f467a854f0e49c4985bedcc0cb93479. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:12.191005 9273 tablet_bootstrap.cc:492] T 0f467a854f0e49c4985bedcc0cb93479 P 03d2d4fa8a0d429e82d7a36943172f08: Bootstrap starting.
I20260619 08:19:12.191557 9273 tablet_bootstrap.cc:654] T 0f467a854f0e49c4985bedcc0cb93479 P 03d2d4fa8a0d429e82d7a36943172f08: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:12.192062 9273 log.cc:824] T 0f467a854f0e49c4985bedcc0cb93479 P 03d2d4fa8a0d429e82d7a36943172f08: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:12.192112 9274 tablet_bootstrap.cc:492] T 0f467a854f0e49c4985bedcc0cb93479 P 599243d5b3144521911affeebc99c92b: Bootstrap starting.
I20260619 08:19:12.192690 9274 tablet_bootstrap.cc:654] T 0f467a854f0e49c4985bedcc0cb93479 P 599243d5b3144521911affeebc99c92b: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:12.192831 9273 tablet_bootstrap.cc:492] T 0f467a854f0e49c4985bedcc0cb93479 P 03d2d4fa8a0d429e82d7a36943172f08: No bootstrap required, opened a new log
I20260619 08:19:12.192902 9273 ts_tablet_manager.cc:1397] T 0f467a854f0e49c4985bedcc0cb93479 P 03d2d4fa8a0d429e82d7a36943172f08: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:19:12.193282 9274 log.cc:824] T 0f467a854f0e49c4985bedcc0cb93479 P 599243d5b3144521911affeebc99c92b: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:12.193478 9276 tablet_bootstrap.cc:492] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0: Bootstrap starting.
I20260619 08:19:12.194038 9276 tablet_bootstrap.cc:654] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:12.194640 9276 log.cc:824] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:12.194950 9273 raft_consensus.cc:348] T 0f467a854f0e49c4985bedcc0cb93479 P 03d2d4fa8a0d429e82d7a36943172f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599243d5b3144521911affeebc99c92b" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40319 } } peers { permanent_uuid: "33f33cf4204a4962b6714bc0487b7bf0" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 46517 } } peers { permanent_uuid: "03d2d4fa8a0d429e82d7a36943172f08" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41031 } }
I20260619 08:19:12.195073 9273 raft_consensus.cc:374] T 0f467a854f0e49c4985bedcc0cb93479 P 03d2d4fa8a0d429e82d7a36943172f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:12.195118 9273 raft_consensus.cc:729] T 0f467a854f0e49c4985bedcc0cb93479 P 03d2d4fa8a0d429e82d7a36943172f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03d2d4fa8a0d429e82d7a36943172f08, State: Initialized, Role: FOLLOWER
I20260619 08:19:12.195235 9273 consensus_queue.cc:260] T 0f467a854f0e49c4985bedcc0cb93479 P 03d2d4fa8a0d429e82d7a36943172f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599243d5b3144521911affeebc99c92b" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40319 } } peers { permanent_uuid: "33f33cf4204a4962b6714bc0487b7bf0" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 46517 } } peers { permanent_uuid: "03d2d4fa8a0d429e82d7a36943172f08" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41031 } }
I20260619 08:19:12.195334 9276 tablet_bootstrap.cc:492] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0: No bootstrap required, opened a new log
I20260619 08:19:12.195425 9276 ts_tablet_manager.cc:1397] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 08:19:12.195518 8990 heartbeater.cc:503] Master 127.4.151.190:43591 was elected leader, sending a full tablet report...
I20260619 08:19:12.195776 9273 ts_tablet_manager.cc:1428] T 0f467a854f0e49c4985bedcc0cb93479 P 03d2d4fa8a0d429e82d7a36943172f08: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:19:12.196421 9274 tablet_bootstrap.cc:492] T 0f467a854f0e49c4985bedcc0cb93479 P 599243d5b3144521911affeebc99c92b: No bootstrap required, opened a new log
I20260619 08:19:12.196538 9274 ts_tablet_manager.cc:1397] T 0f467a854f0e49c4985bedcc0cb93479 P 599243d5b3144521911affeebc99c92b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 08:19:12.197285 9276 raft_consensus.cc:348] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599243d5b3144521911affeebc99c92b" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40319 } } peers { permanent_uuid: "33f33cf4204a4962b6714bc0487b7bf0" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 46517 } } peers { permanent_uuid: "03d2d4fa8a0d429e82d7a36943172f08" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41031 } }
I20260619 08:19:12.197495 9276 raft_consensus.cc:374] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:12.197546 9276 raft_consensus.cc:729] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33f33cf4204a4962b6714bc0487b7bf0, State: Initialized, Role: FOLLOWER
I20260619 08:19:12.197671 9276 consensus_queue.cc:260] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599243d5b3144521911affeebc99c92b" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40319 } } peers { permanent_uuid: "33f33cf4204a4962b6714bc0487b7bf0" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 46517 } } peers { permanent_uuid: "03d2d4fa8a0d429e82d7a36943172f08" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41031 } }
I20260619 08:19:12.197950 9276 ts_tablet_manager.cc:1428] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260619 08:19:12.197964 9123 heartbeater.cc:503] Master 127.4.151.190:43591 was elected leader, sending a full tablet report...
I20260619 08:19:12.198618 9274 raft_consensus.cc:348] T 0f467a854f0e49c4985bedcc0cb93479 P 599243d5b3144521911affeebc99c92b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599243d5b3144521911affeebc99c92b" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40319 } } peers { permanent_uuid: "33f33cf4204a4962b6714bc0487b7bf0" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 46517 } } peers { permanent_uuid: "03d2d4fa8a0d429e82d7a36943172f08" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41031 } }
I20260619 08:19:12.198773 9274 raft_consensus.cc:374] T 0f467a854f0e49c4985bedcc0cb93479 P 599243d5b3144521911affeebc99c92b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:12.198817 9274 raft_consensus.cc:729] T 0f467a854f0e49c4985bedcc0cb93479 P 599243d5b3144521911affeebc99c92b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599243d5b3144521911affeebc99c92b, State: Initialized, Role: FOLLOWER
I20260619 08:19:12.198925 9274 consensus_queue.cc:260] T 0f467a854f0e49c4985bedcc0cb93479 P 599243d5b3144521911affeebc99c92b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599243d5b3144521911affeebc99c92b" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40319 } } peers { permanent_uuid: "33f33cf4204a4962b6714bc0487b7bf0" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 46517 } } peers { permanent_uuid: "03d2d4fa8a0d429e82d7a36943172f08" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41031 } }
I20260619 08:19:12.199246 9274 ts_tablet_manager.cc:1428] T 0f467a854f0e49c4985bedcc0cb93479 P 599243d5b3144521911affeebc99c92b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:19:12.199348 9256 heartbeater.cc:503] Master 127.4.151.190:43591 was elected leader, sending a full tablet report...
W20260619 08:19:12.250929 9124 tablet.cc:2369] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:19:12.307381 9280 raft_consensus.cc:482] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:19:12.307523 9280 raft_consensus.cc:504] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599243d5b3144521911affeebc99c92b" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40319 } } peers { permanent_uuid: "33f33cf4204a4962b6714bc0487b7bf0" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 46517 } } peers { permanent_uuid: "03d2d4fa8a0d429e82d7a36943172f08" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41031 } }
I20260619 08:19:12.307942 9280 leader_election.cc:288] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 599243d5b3144521911affeebc99c92b (127.4.151.131:40319), 03d2d4fa8a0d429e82d7a36943172f08 (127.4.151.129:41031)
I20260619 08:19:12.312201 9125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.308148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:40319 (local address 127.4.151.130:45921)
0619 08:19:12.308288 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:12.308301 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:12.308362 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:12.308556 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:12.308557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:12.308559 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:12.308656 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:12.308658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.309622 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.309624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:12.311014 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:12.311018 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.311870 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.311871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:12.311966 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:12.311970 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:19:12.312036 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 08:19:12.312096 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":27}
I20260619 08:19:12.312227 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.308175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:45921 (local address 127.4.151.131:40319)
0619 08:19:12.308251 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:12.308253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:12.308420 (+ 167us) server_negotiation.cc:407] Connection header received
0619 08:19:12.308458 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:12.308459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:12.308471 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:12.308529 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:12.308735 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.309537 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.311129 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.311770 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.311848 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:12.311858 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:12.312107 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 08:19:12.312157 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 08:19:12.312239 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.308393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:43549 (local address 127.4.151.129:41031)
0619 08:19:12.308446 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:12.308448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:12.308500 (+ 52us) server_negotiation.cc:407] Connection header received
0619 08:19:12.308589 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:12.308590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:12.308601 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:12.308647 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:12.308813 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.309627 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.311210 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.311763 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.311848 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:12.311938 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:12.312133 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 08:19:12.312181 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 08:19:12.312208 9282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.308251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:41031 (local address 127.4.151.130:43549)
0619 08:19:12.308435 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:12.308444 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:12.308523 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:12.308674 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:12.308675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:12.308677 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:12.308736 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:12.308737 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.309811 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.309811 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:12.311111 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:12.311113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.311870 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.311871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:12.311963 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:12.312047 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 08:19:12.312101 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 08:19:12.312129 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":3,"thread_start_us":119,"threads_started":1}
I20260619 08:19:12.312589 9210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f467a854f0e49c4985bedcc0cb93479" candidate_uuid: "33f33cf4204a4962b6714bc0487b7bf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "599243d5b3144521911affeebc99c92b" is_pre_election: true
I20260619 08:19:12.312594 8944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f467a854f0e49c4985bedcc0cb93479" candidate_uuid: "33f33cf4204a4962b6714bc0487b7bf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d2d4fa8a0d429e82d7a36943172f08" is_pre_election: true
I20260619 08:19:12.312753 8944 raft_consensus.cc:2454] T 0f467a854f0e49c4985bedcc0cb93479 P 03d2d4fa8a0d429e82d7a36943172f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33f33cf4204a4962b6714bc0487b7bf0 in term 0.
I20260619 08:19:12.312759 9210 raft_consensus.cc:2454] T 0f467a854f0e49c4985bedcc0cb93479 P 599243d5b3144521911affeebc99c92b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33f33cf4204a4962b6714bc0487b7bf0 in term 0.
I20260619 08:19:12.313007 9011 leader_election.cc:302] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03d2d4fa8a0d429e82d7a36943172f08, 33f33cf4204a4962b6714bc0487b7bf0; no voters:
I20260619 08:19:12.313148 9280 raft_consensus.cc:2781] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:19:12.313206 9280 raft_consensus.cc:482] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:19:12.313234 9280 raft_consensus.cc:3037] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:12.313966 9280 raft_consensus.cc:504] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599243d5b3144521911affeebc99c92b" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40319 } } peers { permanent_uuid: "33f33cf4204a4962b6714bc0487b7bf0" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 46517 } } peers { permanent_uuid: "03d2d4fa8a0d429e82d7a36943172f08" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41031 } }
I20260619 08:19:12.314129 9280 leader_election.cc:288] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0 [CANDIDATE]: Term 1 election: Requested vote from peers 599243d5b3144521911affeebc99c92b (127.4.151.131:40319), 03d2d4fa8a0d429e82d7a36943172f08 (127.4.151.129:41031)
I20260619 08:19:12.314278 9210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f467a854f0e49c4985bedcc0cb93479" candidate_uuid: "33f33cf4204a4962b6714bc0487b7bf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "599243d5b3144521911affeebc99c92b"
I20260619 08:19:12.314314 8944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f467a854f0e49c4985bedcc0cb93479" candidate_uuid: "33f33cf4204a4962b6714bc0487b7bf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d2d4fa8a0d429e82d7a36943172f08"
I20260619 08:19:12.314385 8944 raft_consensus.cc:3037] T 0f467a854f0e49c4985bedcc0cb93479 P 03d2d4fa8a0d429e82d7a36943172f08 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:12.314406 9210 raft_consensus.cc:3037] T 0f467a854f0e49c4985bedcc0cb93479 P 599243d5b3144521911affeebc99c92b [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:12.315114 8944 raft_consensus.cc:2454] T 0f467a854f0e49c4985bedcc0cb93479 P 03d2d4fa8a0d429e82d7a36943172f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33f33cf4204a4962b6714bc0487b7bf0 in term 1.
I20260619 08:19:12.315277 9210 raft_consensus.cc:2454] T 0f467a854f0e49c4985bedcc0cb93479 P 599243d5b3144521911affeebc99c92b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33f33cf4204a4962b6714bc0487b7bf0 in term 1.
I20260619 08:19:12.315294 9011 leader_election.cc:302] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03d2d4fa8a0d429e82d7a36943172f08, 33f33cf4204a4962b6714bc0487b7bf0; no voters:
I20260619 08:19:12.315387 9280 raft_consensus.cc:2781] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:12.315548 9280 raft_consensus.cc:686] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0 [term 1 LEADER]: Becoming Leader. State: Replica: 33f33cf4204a4962b6714bc0487b7bf0, State: Running, Role: LEADER
I20260619 08:19:12.315658 9280 consensus_queue.cc:237] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599243d5b3144521911affeebc99c92b" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40319 } } peers { permanent_uuid: "33f33cf4204a4962b6714bc0487b7bf0" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 46517 } } peers { permanent_uuid: "03d2d4fa8a0d429e82d7a36943172f08" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41031 } }
I20260619 08:19:12.316524 8800 catalog_manager.cc:5495] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33f33cf4204a4962b6714bc0487b7bf0 (127.4.151.130). New cstate: current_term: 1 leader_uuid: "33f33cf4204a4962b6714bc0487b7bf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599243d5b3144521911affeebc99c92b" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33f33cf4204a4962b6714bc0487b7bf0" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 46517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03d2d4fa8a0d429e82d7a36943172f08" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41031 } health_report { overall_health: UNKNOWN } } }
I20260619 08:19:12.332583 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.330113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37816 (local address 127.4.151.130:46517)
0619 08:19:12.330168 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:12.330170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:12.330180 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:19:12.330255 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:12.330256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:12.330272 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:12.330328 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:19:12.330480 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.330985 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.331822 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:12.332021 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:12.332083 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:12.332238 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:19:12.332356 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:19:12.332427 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:12.332482 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 08:19:12.332501 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 08:19:12.334730 8944 raft_consensus.cc:1261] T 0f467a854f0e49c4985bedcc0cb93479 P 03d2d4fa8a0d429e82d7a36943172f08 [term 1 FOLLOWER]: Refusing update from remote peer 33f33cf4204a4962b6714bc0487b7bf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:19:12.334730 9210 raft_consensus.cc:1261] T 0f467a854f0e49c4985bedcc0cb93479 P 599243d5b3144521911affeebc99c92b [term 1 FOLLOWER]: Refusing update from remote peer 33f33cf4204a4962b6714bc0487b7bf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:19:12.335172 9283 consensus_queue.cc:1035] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03d2d4fa8a0d429e82d7a36943172f08" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:19:12.335323 9280 consensus_queue.cc:1035] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "599243d5b3144521911affeebc99c92b" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:19:12.337739 9286 mvcc.cc:205] Tried to move back new op lower bound from 7298486895959785472 to 7298486895885479936. Current Snapshot: MvccSnapshot[applied={T|T < 7298486895959785472}]
W20260619 08:19:12.337904 8991 tablet.cc:2369] T 0f467a854f0e49c4985bedcc0cb93479 P 03d2d4fa8a0d429e82d7a36943172f08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:19:12.338104 9288 mvcc.cc:205] Tried to move back new op lower bound from 7298486895959785472 to 7298486895885479936. Current Snapshot: MvccSnapshot[applied={T|T < 7298486895959785472}]
I20260619 08:19:12.340039 8799 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:19:12.340157 8799 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:19:12.340916 8799 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b103b400de644931826acced764cec3b: Sending DeleteTablet for 3 replicas of tablet 0f467a854f0e49c4985bedcc0cb93479
I20260619 08:19:12.341282 9190 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f467a854f0e49c4985bedcc0cb93479 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:12 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:39206
I20260619 08:19:12.341526 9296 tablet_replica.cc:330] T 0f467a854f0e49c4985bedcc0cb93479 P 599243d5b3144521911affeebc99c92b: stopping tablet replica
I20260619 08:19:12.341528 9057 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f467a854f0e49c4985bedcc0cb93479 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:12 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:37804
I20260619 08:19:12.341631 9296 raft_consensus.cc:2229] T 0f467a854f0e49c4985bedcc0cb93479 P 599243d5b3144521911affeebc99c92b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:19:12.341750 9296 raft_consensus.cc:2258] T 0f467a854f0e49c4985bedcc0cb93479 P 599243d5b3144521911affeebc99c92b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:12.341768 9297 tablet_replica.cc:330] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0: stopping tablet replica
I20260619 08:19:12.341889 8924 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f467a854f0e49c4985bedcc0cb93479 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:12 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:51218
I20260619 08:19:12.341945 9297 raft_consensus.cc:2229] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:19:12.342147 9297 raft_consensus.cc:2258] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:12.342224 9296 ts_tablet_manager.cc:1905] T 0f467a854f0e49c4985bedcc0cb93479 P 599243d5b3144521911affeebc99c92b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:12.342373 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 8862
I20260619 08:19:12.343024 9297 ts_tablet_manager.cc:1905] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:12.344533 9296 ts_tablet_manager.cc:1918] T 0f467a854f0e49c4985bedcc0cb93479 P 599243d5b3144521911affeebc99c92b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:19:12.344614 9296 log.cc:1196] T 0f467a854f0e49c4985bedcc0cb93479 P 599243d5b3144521911affeebc99c92b: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/0f467a854f0e49c4985bedcc0cb93479
I20260619 08:19:12.344892 9297 ts_tablet_manager.cc:1918] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:19:12.344947 9296 ts_tablet_manager.cc:1939] T 0f467a854f0e49c4985bedcc0cb93479 P 599243d5b3144521911affeebc99c92b: Deleting consensus metadata
I20260619 08:19:12.345050 9297 log.cc:1196] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/0f467a854f0e49c4985bedcc0cb93479
I20260619 08:19:12.345358 8787 catalog_manager.cc:4841] TS 599243d5b3144521911affeebc99c92b (127.4.151.131:40319): tablet 0f467a854f0e49c4985bedcc0cb93479 (table test-table [id=be1949e040144be48e13d94a614f4c75]) successfully deleted
I20260619 08:19:12.345431 9297 ts_tablet_manager.cc:1939] T 0f467a854f0e49c4985bedcc0cb93479 P 33f33cf4204a4962b6714bc0487b7bf0: Deleting consensus metadata
I20260619 08:19:12.345917 8786 catalog_manager.cc:4841] TS 33f33cf4204a4962b6714bc0487b7bf0 (127.4.151.130:46517): tablet 0f467a854f0e49c4985bedcc0cb93479 (table test-table [id=be1949e040144be48e13d94a614f4c75]) successfully deleted
W20260619 08:19:12.349212 8785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.151.129:41031 (error 108)
I20260619 08:19:12.349468 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 8995
I20260619 08:19:12.350297 9269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.350156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:41031 (local address 127.0.0.1:51228)
0619 08:19:12.350217 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:12.350256 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.129:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 08:19:12.350466 8785 catalog_manager.cc:4584] TS 03d2d4fa8a0d429e82d7a36943172f08 (127.4.151.129:41031): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f467a854f0e49c4985bedcc0cb93479: Network error: Client connection negotiation failed: client connection to 127.4.151.129:41031: connect: Connection refused (error 111)
I20260619 08:19:12.354907 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 9128
I20260619 08:19:12.359552 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 8769
2026-06-19T08:19:12Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2007 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T08:19:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:19:12Z Disabled control of system clock
I20260619 08:19:12.387806 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:38397
--webserver_interface=127.4.151.190
--webserver_port=0
--builtin_ntp_servers=127.4.151.148:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:38397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 08:19:12.469149 9303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:12.469394 9303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:12.469416 9303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:12.470614 9303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:19:12.470656 9303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:12.470669 9303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:12.470682 9303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:19:12.470695 9303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:19:12.472209 9303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:38397
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:38397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:12.472486 9303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:12.472791 9303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:12.477979 9309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:12.477941 9311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:12.478070 9308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:12.478195 9303 server_base.cc:1034] running on GCE node
I20260619 08:19:12.478536 9303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:12.478794 9303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:12.479964 9303 hybrid_clock.cc:648] HybridClock initialized: now 1781857152479940 us; error 56 us; skew 500 ppm
I20260619 08:19:12.481348 9303 webserver.cc:460] Webserver started at http://127.4.151.190:36077/ using document root <none> and password file <none>
I20260619 08:19:12.481673 9303 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:12.481743 9303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:12.481899 9303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:12.483027 9303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "d25679ead9c847a0a87d48b31fcf3417"
format_stamp: "Formatted at 2026-06-19 08:19:12 on dist-test-slave-d3n8"
server_key: "eda9a4ca8540ee0410e3adb6bbcd0246"
server_key_iv: "0c538350270938536491782c204b7c39"
server_key_version: "encryptionkey@0"
I20260619 08:19:12.483405 9303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "d25679ead9c847a0a87d48b31fcf3417"
format_stamp: "Formatted at 2026-06-19 08:19:12 on dist-test-slave-d3n8"
server_key: "eda9a4ca8540ee0410e3adb6bbcd0246"
server_key_iv: "0c538350270938536491782c204b7c39"
server_key_version: "encryptionkey@0"
I20260619 08:19:12.484758 9303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:19:12.485615 9317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:12.485908 9303 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:12.486006 9303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "d25679ead9c847a0a87d48b31fcf3417"
format_stamp: "Formatted at 2026-06-19 08:19:12 on dist-test-slave-d3n8"
server_key: "eda9a4ca8540ee0410e3adb6bbcd0246"
server_key_iv: "0c538350270938536491782c204b7c39"
server_key_version: "encryptionkey@0"
I20260619 08:19:12.486124 9303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:12.489630 9303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:12.489915 9303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:12.490060 9303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:12.494252 9303 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:38397
I20260619 08:19:12.494283 9369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:38397 every 8 connection(s)
I20260619 08:19:12.494618 9303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:19:12.495287 9370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:12.498045 9370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d25679ead9c847a0a87d48b31fcf3417: Bootstrap starting.
I20260619 08:19:12.498572 9370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d25679ead9c847a0a87d48b31fcf3417: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:12.498821 9370 log.cc:824] T 00000000000000000000000000000000 P d25679ead9c847a0a87d48b31fcf3417: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:12.499526 9370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d25679ead9c847a0a87d48b31fcf3417: No bootstrap required, opened a new log
I20260619 08:19:12.501076 9370 raft_consensus.cc:348] T 00000000000000000000000000000000 P d25679ead9c847a0a87d48b31fcf3417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25679ead9c847a0a87d48b31fcf3417" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38397 } }
I20260619 08:19:12.501216 9370 raft_consensus.cc:374] T 00000000000000000000000000000000 P d25679ead9c847a0a87d48b31fcf3417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:12.501238 9370 raft_consensus.cc:729] T 00000000000000000000000000000000 P d25679ead9c847a0a87d48b31fcf3417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d25679ead9c847a0a87d48b31fcf3417, State: Initialized, Role: FOLLOWER
I20260619 08:19:12.501384 9370 consensus_queue.cc:260] T 00000000000000000000000000000000 P d25679ead9c847a0a87d48b31fcf3417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25679ead9c847a0a87d48b31fcf3417" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38397 } }
I20260619 08:19:12.501497 9370 raft_consensus.cc:388] T 00000000000000000000000000000000 P d25679ead9c847a0a87d48b31fcf3417 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:19:12.501519 9370 raft_consensus.cc:482] T 00000000000000000000000000000000 P d25679ead9c847a0a87d48b31fcf3417 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:19:12.501549 9370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d25679ead9c847a0a87d48b31fcf3417 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:12.502104 9370 raft_consensus.cc:504] T 00000000000000000000000000000000 P d25679ead9c847a0a87d48b31fcf3417 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25679ead9c847a0a87d48b31fcf3417" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38397 } }
I20260619 08:19:12.502223 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 9303
I20260619 08:19:12.502223 9370 leader_election.cc:302] T 00000000000000000000000000000000 P d25679ead9c847a0a87d48b31fcf3417 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d25679ead9c847a0a87d48b31fcf3417; no voters:
I20260619 08:19:12.502328 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:19:12.502395 9370 leader_election.cc:288] T 00000000000000000000000000000000 P d25679ead9c847a0a87d48b31fcf3417 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:19:12.502487 9373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d25679ead9c847a0a87d48b31fcf3417 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:12.502552 4702 external_mini_cluster.cc:1428] Setting key c7838ee0af6ac42e3ac9879c91e7286c
I20260619 08:19:12.502678 9373 raft_consensus.cc:686] T 00000000000000000000000000000000 P d25679ead9c847a0a87d48b31fcf3417 [term 1 LEADER]: Becoming Leader. State: Replica: d25679ead9c847a0a87d48b31fcf3417, State: Running, Role: LEADER
I20260619 08:19:12.502692 9370 sys_catalog.cc:564] T 00000000000000000000000000000000 P d25679ead9c847a0a87d48b31fcf3417 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:19:12.502791 9373 consensus_queue.cc:237] T 00000000000000000000000000000000 P d25679ead9c847a0a87d48b31fcf3417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25679ead9c847a0a87d48b31fcf3417" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38397 } }
I20260619 08:19:12.503356 9374 sys_catalog.cc:455] T 00000000000000000000000000000000 P d25679ead9c847a0a87d48b31fcf3417 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d25679ead9c847a0a87d48b31fcf3417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25679ead9c847a0a87d48b31fcf3417" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38397 } } }
I20260619 08:19:12.503445 9374 sys_catalog.cc:458] T 00000000000000000000000000000000 P d25679ead9c847a0a87d48b31fcf3417 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:12.503602 9375 sys_catalog.cc:455] T 00000000000000000000000000000000 P d25679ead9c847a0a87d48b31fcf3417 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d25679ead9c847a0a87d48b31fcf3417. Latest consensus state: current_term: 1 leader_uuid: "d25679ead9c847a0a87d48b31fcf3417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25679ead9c847a0a87d48b31fcf3417" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38397 } } }
I20260619 08:19:12.503674 9375 sys_catalog.cc:458] T 00000000000000000000000000000000 P d25679ead9c847a0a87d48b31fcf3417 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:12.503794 9384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:19:12.504357 9384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:19:12.506001 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.503190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37316 (local address 127.4.151.190:38397)
0619 08:19:12.503654 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:12.503665 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:12.503876 (+ 211us) server_negotiation.cc:407] Connection header received
0619 08:19:12.504022 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:12.504031 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:12.504116 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:12.505155 (+ 1039us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:12.505533 (+ 378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:12.505535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:12.505536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:19:12.505539 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:12.505581 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:19:12.505594 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:12.505668 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:12.505778 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 08:19:12.505841 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":59,"threads_started":1}
I20260619 08:19:12.506319 9384 catalog_manager.cc:1347] Generated new cluster ID: 31a3cb60f70f4e76a18d84878220f2b0
I20260619 08:19:12.506388 9384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:19:12.516176 9384 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:19:12.516396 9384 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:19:12.526348 9384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d25679ead9c847a0a87d48b31fcf3417: Generated new TSK 0
I20260619 08:19:12.526624 9384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:19:12.530746 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:38397
--builtin_ntp_servers=127.4.151.148:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 08:19:12.612576 9394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:12.612815 9394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:12.612855 9394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:12.614056 9394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:12.614125 9394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:12.614171 9394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:19:12.615626 9394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:38397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:12.615911 9394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:12.616211 9394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:19:12.622157 9394 server_base.cc:1034] running on GCE node
W20260619 08:19:12.622171 9400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:12.622272 9399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:12.622277 9402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:12.622641 9394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:12.622901 9394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:12.624068 9394 hybrid_clock.cc:648] HybridClock initialized: now 1781857152624051 us; error 27 us; skew 500 ppm
I20260619 08:19:12.625623 9394 webserver.cc:460] Webserver started at http://127.4.151.129:45925/ using document root <none> and password file <none>
I20260619 08:19:12.625895 9394 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:12.625960 9394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:12.626094 9394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:12.627501 9394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "8a95e1aebd794fb2ac2e3b0c705090e7"
format_stamp: "Formatted at 2026-06-19 08:19:12 on dist-test-slave-d3n8"
server_key: "540612215e38f6500e25b18ed013b047"
server_key_iv: "6743a378e491f25c9f170c105a6416df"
server_key_version: "encryptionkey@0"
I20260619 08:19:12.627902 9394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "8a95e1aebd794fb2ac2e3b0c705090e7"
format_stamp: "Formatted at 2026-06-19 08:19:12 on dist-test-slave-d3n8"
server_key: "540612215e38f6500e25b18ed013b047"
server_key_iv: "6743a378e491f25c9f170c105a6416df"
server_key_version: "encryptionkey@0"
I20260619 08:19:12.629788 9394 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260619 08:19:12.630714 9408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:12.631008 9394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:12.631110 9394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "8a95e1aebd794fb2ac2e3b0c705090e7"
format_stamp: "Formatted at 2026-06-19 08:19:12 on dist-test-slave-d3n8"
server_key: "540612215e38f6500e25b18ed013b047"
server_key_iv: "6743a378e491f25c9f170c105a6416df"
server_key_version: "encryptionkey@0"
I20260619 08:19:12.631191 9394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:12.634649 9394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:12.634920 9394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:12.635064 9394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:12.635263 9394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:12.635624 9394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:12.635665 9394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:12.635685 9394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:12.635720 9394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:12.641420 9394 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:36125
I20260619 08:19:12.641506 9521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:36125 every 8 connection(s)
I20260619 08:19:12.641816 9394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
I20260619 08:19:12.644899 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.642528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:56953 (local address 127.4.151.190:38397)
0619 08:19:12.642622 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:12.642624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:12.642822 (+ 198us) server_negotiation.cc:407] Connection header received
0619 08:19:12.643350 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:12.643352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:12.643360 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:12.643411 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:12.644593 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:12.644594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:12.644595 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:19:12.644601 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:12.644617 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:19:12.644622 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:12.644702 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:12.644811 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 08:19:12.644829 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 08:19:12.645138 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.642508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:38397 (local address 127.4.151.129:56953)
0619 08:19:12.642702 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:12.642736 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:12.643233 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:12.643491 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:12.643494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:12.643501 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:12.644418 (+ 917us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:19:12.644418 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:12.644453 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:19:12.644480 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:12.644699 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:12.644705 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:12.644708 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:19:12.644817 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 08:19:12.644880 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260619 08:19:12.645794 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 9394
I20260619 08:19:12.645812 9522 heartbeater.cc:347] Connected to a master server at 127.4.151.190:38397
I20260619 08:19:12.645910 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:19:12.645921 9522 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:12.646061 9522 heartbeater.cc:511] Master 127.4.151.190:38397 requested a full tablet report, sending...
I20260619 08:19:12.646112 4702 external_mini_cluster.cc:1428] Setting key 7e2c380b7412dc7a240f9ba4fa399a6d
I20260619 08:19:12.646661 9333 ts_manager.cc:194] Registered new tserver with Master: 8a95e1aebd794fb2ac2e3b0c705090e7 (127.4.151.129:36125)
I20260619 08:19:12.647025 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:38397
--builtin_ntp_servers=127.4.151.148:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 08:19:12.728051 9525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:12.728266 9525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:12.728289 9525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:12.729532 9525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:12.729578 9525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:12.729604 9525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:19:12.730957 9525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:38397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:12.731226 9525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:12.731554 9525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:12.736845 9533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:12.736937 9530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:12.736845 9531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:12.737252 9525 server_base.cc:1034] running on GCE node
I20260619 08:19:12.737504 9525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:12.737756 9525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:12.738940 9525 hybrid_clock.cc:648] HybridClock initialized: now 1781857152738915 us; error 34 us; skew 500 ppm
I20260619 08:19:12.740262 9525 webserver.cc:460] Webserver started at http://127.4.151.130:41329/ using document root <none> and password file <none>
I20260619 08:19:12.740572 9525 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:12.740643 9525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:12.740772 9525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:12.741977 9525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "bd6378f8fa624434a9a7c65b1de75818"
format_stamp: "Formatted at 2026-06-19 08:19:12 on dist-test-slave-d3n8"
server_key: "b0539a5dd72a450e7d08bb60f717c58c"
server_key_iv: "6ed22e044c20af3b876c0ce27616c1ec"
server_key_version: "encryptionkey@0"
I20260619 08:19:12.742300 9525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "bd6378f8fa624434a9a7c65b1de75818"
format_stamp: "Formatted at 2026-06-19 08:19:12 on dist-test-slave-d3n8"
server_key: "b0539a5dd72a450e7d08bb60f717c58c"
server_key_iv: "6ed22e044c20af3b876c0ce27616c1ec"
server_key_version: "encryptionkey@0"
I20260619 08:19:12.743571 9525 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 08:19:12.744287 9539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:12.744483 9525 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:12.744556 9525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "bd6378f8fa624434a9a7c65b1de75818"
format_stamp: "Formatted at 2026-06-19 08:19:12 on dist-test-slave-d3n8"
server_key: "b0539a5dd72a450e7d08bb60f717c58c"
server_key_iv: "6ed22e044c20af3b876c0ce27616c1ec"
server_key_version: "encryptionkey@0"
I20260619 08:19:12.744608 9525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:12.747968 9525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:12.748201 9525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:12.748296 9525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:12.748445 9525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:12.748747 9525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:12.748780 9525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:12.748800 9525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:12.748847 9525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:12.754733 9525 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:37227
I20260619 08:19:12.754822 9652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:37227 every 8 connection(s)
I20260619 08:19:12.755122 9525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
I20260619 08:19:12.758481 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.755849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:45455 (local address 127.4.151.190:38397)
0619 08:19:12.755899 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:12.755901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:12.756158 (+ 257us) server_negotiation.cc:407] Connection header received
0619 08:19:12.756654 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:12.756656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:12.756662 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:12.756731 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:12.758203 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:12.758204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:12.758205 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:19:12.758209 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:12.758226 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:19:12.758231 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:12.758290 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:12.758412 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 08:19:12.758425 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 08:19:12.758678 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.755835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:38397 (local address 127.4.151.130:45455)
0619 08:19:12.756023 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:12.756063 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:12.756539 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:12.756830 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:12.756842 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:12.756929 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:12.758041 (+ 1112us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:19:12.758042 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:12.758077 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:19:12.758106 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:12.758323 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:12.758329 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:12.758332 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:19:12.758413 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 08:19:12.758466 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260619 08:19:12.759204 9653 heartbeater.cc:347] Connected to a master server at 127.4.151.190:38397
I20260619 08:19:12.759299 9653 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:12.759418 9653 heartbeater.cc:511] Master 127.4.151.190:38397 requested a full tablet report, sending...
I20260619 08:19:12.759814 9333 ts_manager.cc:194] Registered new tserver with Master: bd6378f8fa624434a9a7c65b1de75818 (127.4.151.130:37227)
I20260619 08:19:12.761947 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 9525
I20260619 08:19:12.762025 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:19:12.762163 4702 external_mini_cluster.cc:1428] Setting key 9a79b077fd006f245722914add3defa6
I20260619 08:19:12.763190 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:38397
--builtin_ntp_servers=127.4.151.148:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 08:19:12.847023 9656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:12.847278 9656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:12.847319 9656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:12.848649 9656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:12.848717 9656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:12.848760 9656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:19:12.850198 9656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:38397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:12.850493 9656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:12.850792 9656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:12.856393 9664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:12.856401 9661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:12.856401 9662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:12.856640 9656 server_base.cc:1034] running on GCE node
I20260619 08:19:12.856940 9656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:12.857215 9656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:12.858366 9656 hybrid_clock.cc:648] HybridClock initialized: now 1781857152858348 us; error 37 us; skew 500 ppm
I20260619 08:19:12.859635 9656 webserver.cc:460] Webserver started at http://127.4.151.131:43691/ using document root <none> and password file <none>
I20260619 08:19:12.859907 9656 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:12.859977 9656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:12.860122 9656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:12.861243 9656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "74beda5e6bdb4336a99c7df448188859"
format_stamp: "Formatted at 2026-06-19 08:19:12 on dist-test-slave-d3n8"
server_key: "06fe1a7f8407cf3c97619770e6f1659e"
server_key_iv: "c932abcb4b8ba924425aaf3423113e8d"
server_key_version: "encryptionkey@0"
I20260619 08:19:12.861663 9656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "74beda5e6bdb4336a99c7df448188859"
format_stamp: "Formatted at 2026-06-19 08:19:12 on dist-test-slave-d3n8"
server_key: "06fe1a7f8407cf3c97619770e6f1659e"
server_key_iv: "c932abcb4b8ba924425aaf3423113e8d"
server_key_version: "encryptionkey@0"
I20260619 08:19:12.863088 9656 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:12.864108 9670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:12.864312 9656 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:12.864410 9656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "74beda5e6bdb4336a99c7df448188859"
format_stamp: "Formatted at 2026-06-19 08:19:12 on dist-test-slave-d3n8"
server_key: "06fe1a7f8407cf3c97619770e6f1659e"
server_key_iv: "c932abcb4b8ba924425aaf3423113e8d"
server_key_version: "encryptionkey@0"
I20260619 08:19:12.864490 9656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:12.867954 9656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:12.868225 9656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:12.868353 9656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:12.868568 9656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:12.868872 9656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:12.868922 9656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:12.868944 9656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:12.868957 9656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:12.875422 9656 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:41079
I20260619 08:19:12.875584 9783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:41079 every 8 connection(s)
I20260619 08:19:12.875830 9656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
I20260619 08:19:12.877924 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 9656
I20260619 08:19:12.878015 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:19:12.878223 4702 external_mini_cluster.cc:1428] Setting key 2cd43055ae2de516bd4bbd5accdb4fb4
I20260619 08:19:12.879582 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.876662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:45993 (local address 127.4.151.190:38397)
0619 08:19:12.876775 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:12.876778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:12.877041 (+ 263us) server_negotiation.cc:407] Connection header received
0619 08:19:12.877639 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:12.877641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:12.877647 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:12.877707 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:12.879289 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:12.879290 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:12.879291 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:19:12.879295 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:12.879309 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:19:12.879314 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:12.879380 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:12.879479 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 08:19:12.879507 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 08:19:12.879778 9786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.876573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:38397 (local address 127.4.151.131:45993)
0619 08:19:12.876836 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:12.876932 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:12.877530 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:12.877804 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:12.877808 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:12.877816 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:12.879051 (+ 1235us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:19:12.879051 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:12.879081 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:19:12.879109 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:12.879379 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:12.879385 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:12.879388 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:19:12.879491 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 08:19:12.879556 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":123,"threads_started":1}
I20260619 08:19:12.880292 9784 heartbeater.cc:347] Connected to a master server at 127.4.151.190:38397
I20260619 08:19:12.880385 9784 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:12.880548 9784 heartbeater.cc:511] Master 127.4.151.190:38397 requested a full tablet report, sending...
I20260619 08:19:12.880890 9333 ts_manager.cc:194] Registered new tserver with Master: 74beda5e6bdb4336a99c7df448188859 (127.4.151.131:41079)
I20260619 08:19:12.890132 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:19:12.892762 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.891525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37320 (local address 127.4.151.190:38397)
0619 08:19:12.891597 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:12.891599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:12.891702 (+ 103us) server_negotiation.cc:407] Connection header received
0619 08:19:12.891759 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:12.891760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:12.891764 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:12.891814 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:12.892489 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:12.892490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:12.892490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:19:12.892494 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:12.892510 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:19:12.892514 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:12.892571 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:12.892625 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 08:19:12.892638 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 08:19:12.894695 9333 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:19:12.896226 9333 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:19:12.900972 9795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.898974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:36125 (local address 127.0.0.1:59424)
0619 08:19:12.899170 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:12.899183 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:12.899239 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:12.899840 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:12.899841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:12.899845 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:12.900504 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:19:12.900505 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:12.900523 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:19:12.900549 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:12.900792 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:12.900798 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:12.900801 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:19:12.900867 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 08:19:12.900882 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":131,"threads_started":1}
I20260619 08:19:12.901014 9796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.898937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49252 (local address 127.4.151.130:37227)
0619 08:19:12.899313 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:12.899318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:12.899328 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:19:12.899354 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:12.899357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:12.899365 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:12.899444 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:12.900666 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:12.900674 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:12.900674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:19:12.900678 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:12.900716 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:19:12.900734 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:12.900805 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:12.900919 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 08:19:12.900933 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":185,"threads_started":1}
I20260619 08:19:12.900988 9797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.899247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59424 (local address 127.4.151.129:36125)
0619 08:19:12.899434 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:12.899438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:12.899447 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:19:12.899471 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:12.899474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:12.899481 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:12.899549 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:12.900630 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:12.900640 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:12.900641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:19:12.900646 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:12.900685 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:19:12.900718 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:12.900788 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:12.900884 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 08:19:12.900900 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":88,"threads_started":1}
I20260619 08:19:12.901189 9793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.898748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:41079 (local address 127.0.0.1:49164)
0619 08:19:12.898945 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:12.898989 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:12.899045 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:12.899823 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:12.899828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:12.899834 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:12.900565 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:19:12.900566 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:12.900573 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:19:12.900586 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:12.900810 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:12.900811 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:12.900812 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 08:19:12.900845 (+ 33us) client_negotiation.cc:240] Negotiation successful
0619 08:19:12.900860 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":62,"threads_started":1}
I20260619 08:19:12.901391 9792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.898749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:37227 (local address 127.0.0.1:49252)
0619 08:19:12.899135 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:12.899146 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:12.899180 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:12.899840 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:12.899841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:12.899845 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:12.900504 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:19:12.900505 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:12.900523 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:19:12.900564 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:12.900828 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:12.900829 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:12.900830 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 08:19:12.900874 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 08:19:12.900883 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":29,"thread_start_us":71,"threads_started":1}
I20260619 08:19:12.901397 9794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:12.898850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49164 (local address 127.4.151.131:41079)
0619 08:19:12.899241 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:12.899247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:12.899257 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:19:12.899285 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:12.899287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:12.899297 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:12.899466 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:12.900676 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:12.900682 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:12.900682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:19:12.900686 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:12.900714 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:19:12.900734 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:12.900925 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:12.900946 (+ 21us) server_negotiation.cc:299] Negotiation successful
0619 08:19:12.900961 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":90,"threads_started":1}
I20260619 08:19:12.901953 9456 tablet_service.cc:1467] Processing CreateTablet for tablet ffdef6ab9c1940c49f891bd0ac685225 (DEFAULT_TABLE table=test-table [id=ab79f077fb2f45c7a18c28a44102333b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:12.902024 9587 tablet_service.cc:1467] Processing CreateTablet for tablet ffdef6ab9c1940c49f891bd0ac685225 (DEFAULT_TABLE table=test-table [id=ab79f077fb2f45c7a18c28a44102333b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:12.902024 9718 tablet_service.cc:1467] Processing CreateTablet for tablet ffdef6ab9c1940c49f891bd0ac685225 (DEFAULT_TABLE table=test-table [id=ab79f077fb2f45c7a18c28a44102333b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:12.902374 9718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffdef6ab9c1940c49f891bd0ac685225. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:12.902383 9456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffdef6ab9c1940c49f891bd0ac685225. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:12.902395 9587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffdef6ab9c1940c49f891bd0ac685225. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:12.905354 9798 tablet_bootstrap.cc:492] T ffdef6ab9c1940c49f891bd0ac685225 P 8a95e1aebd794fb2ac2e3b0c705090e7: Bootstrap starting.
I20260619 08:19:12.905993 9799 tablet_bootstrap.cc:492] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818: Bootstrap starting.
I20260619 08:19:12.906109 9798 tablet_bootstrap.cc:654] T ffdef6ab9c1940c49f891bd0ac685225 P 8a95e1aebd794fb2ac2e3b0c705090e7: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:12.906461 9798 log.cc:824] T ffdef6ab9c1940c49f891bd0ac685225 P 8a95e1aebd794fb2ac2e3b0c705090e7: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:12.906682 9799 tablet_bootstrap.cc:654] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:12.907001 9799 log.cc:824] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:12.907285 9798 tablet_bootstrap.cc:492] T ffdef6ab9c1940c49f891bd0ac685225 P 8a95e1aebd794fb2ac2e3b0c705090e7: No bootstrap required, opened a new log
I20260619 08:19:12.907342 9800 tablet_bootstrap.cc:492] T ffdef6ab9c1940c49f891bd0ac685225 P 74beda5e6bdb4336a99c7df448188859: Bootstrap starting.
I20260619 08:19:12.907375 9798 ts_tablet_manager.cc:1397] T ffdef6ab9c1940c49f891bd0ac685225 P 8a95e1aebd794fb2ac2e3b0c705090e7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 08:19:12.907855 9800 tablet_bootstrap.cc:654] T ffdef6ab9c1940c49f891bd0ac685225 P 74beda5e6bdb4336a99c7df448188859: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:12.908139 9800 log.cc:824] T ffdef6ab9c1940c49f891bd0ac685225 P 74beda5e6bdb4336a99c7df448188859: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:12.908686 9799 tablet_bootstrap.cc:492] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818: No bootstrap required, opened a new log
I20260619 08:19:12.908771 9799 ts_tablet_manager.cc:1397] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 08:19:12.908916 9800 tablet_bootstrap.cc:492] T ffdef6ab9c1940c49f891bd0ac685225 P 74beda5e6bdb4336a99c7df448188859: No bootstrap required, opened a new log
I20260619 08:19:12.908994 9800 ts_tablet_manager.cc:1397] T ffdef6ab9c1940c49f891bd0ac685225 P 74beda5e6bdb4336a99c7df448188859: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:19:12.909807 9798 raft_consensus.cc:348] T ffdef6ab9c1940c49f891bd0ac685225 P 8a95e1aebd794fb2ac2e3b0c705090e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74beda5e6bdb4336a99c7df448188859" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41079 } } peers { permanent_uuid: "bd6378f8fa624434a9a7c65b1de75818" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37227 } } peers { permanent_uuid: "8a95e1aebd794fb2ac2e3b0c705090e7" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 36125 } }
I20260619 08:19:12.909986 9798 raft_consensus.cc:374] T ffdef6ab9c1940c49f891bd0ac685225 P 8a95e1aebd794fb2ac2e3b0c705090e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:12.910022 9798 raft_consensus.cc:729] T ffdef6ab9c1940c49f891bd0ac685225 P 8a95e1aebd794fb2ac2e3b0c705090e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a95e1aebd794fb2ac2e3b0c705090e7, State: Initialized, Role: FOLLOWER
I20260619 08:19:12.910233 9798 consensus_queue.cc:260] T ffdef6ab9c1940c49f891bd0ac685225 P 8a95e1aebd794fb2ac2e3b0c705090e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74beda5e6bdb4336a99c7df448188859" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41079 } } peers { permanent_uuid: "bd6378f8fa624434a9a7c65b1de75818" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37227 } } peers { permanent_uuid: "8a95e1aebd794fb2ac2e3b0c705090e7" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 36125 } }
I20260619 08:19:12.910539 9798 ts_tablet_manager.cc:1428] T ffdef6ab9c1940c49f891bd0ac685225 P 8a95e1aebd794fb2ac2e3b0c705090e7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 08:19:12.910655 9522 heartbeater.cc:503] Master 127.4.151.190:38397 was elected leader, sending a full tablet report...
I20260619 08:19:12.911327 9799 raft_consensus.cc:348] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74beda5e6bdb4336a99c7df448188859" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41079 } } peers { permanent_uuid: "bd6378f8fa624434a9a7c65b1de75818" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37227 } } peers { permanent_uuid: "8a95e1aebd794fb2ac2e3b0c705090e7" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 36125 } }
I20260619 08:19:12.911481 9799 raft_consensus.cc:374] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:12.911520 9799 raft_consensus.cc:729] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6378f8fa624434a9a7c65b1de75818, State: Initialized, Role: FOLLOWER
I20260619 08:19:12.911645 9799 consensus_queue.cc:260] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74beda5e6bdb4336a99c7df448188859" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41079 } } peers { permanent_uuid: "bd6378f8fa624434a9a7c65b1de75818" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37227 } } peers { permanent_uuid: "8a95e1aebd794fb2ac2e3b0c705090e7" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 36125 } }
I20260619 08:19:12.911935 9799 ts_tablet_manager.cc:1428] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 08:19:12.911984 9653 heartbeater.cc:503] Master 127.4.151.190:38397 was elected leader, sending a full tablet report...
I20260619 08:19:12.912017 9800 raft_consensus.cc:348] T ffdef6ab9c1940c49f891bd0ac685225 P 74beda5e6bdb4336a99c7df448188859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74beda5e6bdb4336a99c7df448188859" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41079 } } peers { permanent_uuid: "bd6378f8fa624434a9a7c65b1de75818" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37227 } } peers { permanent_uuid: "8a95e1aebd794fb2ac2e3b0c705090e7" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 36125 } }
I20260619 08:19:12.912168 9800 raft_consensus.cc:374] T ffdef6ab9c1940c49f891bd0ac685225 P 74beda5e6bdb4336a99c7df448188859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:12.912206 9800 raft_consensus.cc:729] T ffdef6ab9c1940c49f891bd0ac685225 P 74beda5e6bdb4336a99c7df448188859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74beda5e6bdb4336a99c7df448188859, State: Initialized, Role: FOLLOWER
I20260619 08:19:12.912329 9800 consensus_queue.cc:260] T ffdef6ab9c1940c49f891bd0ac685225 P 74beda5e6bdb4336a99c7df448188859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74beda5e6bdb4336a99c7df448188859" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41079 } } peers { permanent_uuid: "bd6378f8fa624434a9a7c65b1de75818" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37227 } } peers { permanent_uuid: "8a95e1aebd794fb2ac2e3b0c705090e7" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 36125 } }
I20260619 08:19:12.912614 9800 ts_tablet_manager.cc:1428] T ffdef6ab9c1940c49f891bd0ac685225 P 74beda5e6bdb4336a99c7df448188859: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 08:19:12.912691 9784 heartbeater.cc:503] Master 127.4.151.190:38397 was elected leader, sending a full tablet report...
W20260619 08:19:13.005707 9654 tablet.cc:2369] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:19:13.037845 9805 raft_consensus.cc:482] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:19:13.037983 9805 raft_consensus.cc:504] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74beda5e6bdb4336a99c7df448188859" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41079 } } peers { permanent_uuid: "bd6378f8fa624434a9a7c65b1de75818" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37227 } } peers { permanent_uuid: "8a95e1aebd794fb2ac2e3b0c705090e7" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 36125 } }
I20260619 08:19:13.038393 9805 leader_election.cc:288] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74beda5e6bdb4336a99c7df448188859 (127.4.151.131:41079), 8a95e1aebd794fb2ac2e3b0c705090e7 (127.4.151.129:36125)
I20260619 08:19:13.040115 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:13.038701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:41079 (local address 127.4.151.130:41301)
0619 08:19:13.038789 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:13.038803 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:13.038881 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:13.039146 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:13.039148 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:13.039152 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:13.039699 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:19:13.039699 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:13.039707 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:19:13.039722 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:13.039956 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:13.039958 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:13.039959 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 08:19:13.040018 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 08:19:13.040033 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 08:19:13.040161 9794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:13.038824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:41301 (local address 127.4.151.131:41079)
0619 08:19:13.038903 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:13.038906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:13.038918 (+ 12us) server_negotiation.cc:407] Connection header received
0619 08:19:13.039041 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:13.039042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:13.039049 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:13.039107 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:13.039823 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:13.039825 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:13.039825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:19:13.039831 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:13.039851 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:19:13.039856 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:13.039933 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:13.040028 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 08:19:13.040061 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 08:19:13.040220 9797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:13.038961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:43861 (local address 127.4.151.129:36125)
0619 08:19:13.039009 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:13.039011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:13.039140 (+ 129us) server_negotiation.cc:407] Connection header received
0619 08:19:13.039166 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:13.039167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:13.039173 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:13.039231 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:13.039981 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:13.039983 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:13.039983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:19:13.039988 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:13.040006 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:19:13.040011 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:13.040065 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:13.040132 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 08:19:13.040145 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:19:13.040230 9807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:13.038830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:36125 (local address 127.4.151.130:43861)
0619 08:19:13.039026 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:13.039035 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:13.039090 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:13.039262 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:13.039263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:13.039266 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:13.039873 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:19:13.039874 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:13.039880 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:19:13.039893 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:13.040077 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:13.040078 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:13.040079 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 08:19:13.040133 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 08:19:13.040145 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":125,"threads_started":1}
I20260619 08:19:13.040588 9738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffdef6ab9c1940c49f891bd0ac685225" candidate_uuid: "bd6378f8fa624434a9a7c65b1de75818" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74beda5e6bdb4336a99c7df448188859" is_pre_election: true
I20260619 08:19:13.040572 9476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffdef6ab9c1940c49f891bd0ac685225" candidate_uuid: "bd6378f8fa624434a9a7c65b1de75818" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a95e1aebd794fb2ac2e3b0c705090e7" is_pre_election: true
I20260619 08:19:13.040735 9476 raft_consensus.cc:2454] T ffdef6ab9c1940c49f891bd0ac685225 P 8a95e1aebd794fb2ac2e3b0c705090e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd6378f8fa624434a9a7c65b1de75818 in term 0.
I20260619 08:19:13.040735 9738 raft_consensus.cc:2454] T ffdef6ab9c1940c49f891bd0ac685225 P 74beda5e6bdb4336a99c7df448188859 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd6378f8fa624434a9a7c65b1de75818 in term 0.
I20260619 08:19:13.040966 9543 leader_election.cc:302] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74beda5e6bdb4336a99c7df448188859, bd6378f8fa624434a9a7c65b1de75818; no voters:
I20260619 08:19:13.041087 9805 raft_consensus.cc:2781] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:19:13.041146 9805 raft_consensus.cc:482] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:19:13.041172 9805 raft_consensus.cc:3037] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:13.041917 9805 raft_consensus.cc:504] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74beda5e6bdb4336a99c7df448188859" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41079 } } peers { permanent_uuid: "bd6378f8fa624434a9a7c65b1de75818" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37227 } } peers { permanent_uuid: "8a95e1aebd794fb2ac2e3b0c705090e7" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 36125 } }
I20260619 08:19:13.042057 9805 leader_election.cc:288] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818 [CANDIDATE]: Term 1 election: Requested vote from peers 74beda5e6bdb4336a99c7df448188859 (127.4.151.131:41079), 8a95e1aebd794fb2ac2e3b0c705090e7 (127.4.151.129:36125)
I20260619 08:19:13.042241 9738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffdef6ab9c1940c49f891bd0ac685225" candidate_uuid: "bd6378f8fa624434a9a7c65b1de75818" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74beda5e6bdb4336a99c7df448188859"
I20260619 08:19:13.042287 9476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffdef6ab9c1940c49f891bd0ac685225" candidate_uuid: "bd6378f8fa624434a9a7c65b1de75818" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a95e1aebd794fb2ac2e3b0c705090e7"
I20260619 08:19:13.042364 9738 raft_consensus.cc:3037] T ffdef6ab9c1940c49f891bd0ac685225 P 74beda5e6bdb4336a99c7df448188859 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:13.042380 9476 raft_consensus.cc:3037] T ffdef6ab9c1940c49f891bd0ac685225 P 8a95e1aebd794fb2ac2e3b0c705090e7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:13.043313 9738 raft_consensus.cc:2454] T ffdef6ab9c1940c49f891bd0ac685225 P 74beda5e6bdb4336a99c7df448188859 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd6378f8fa624434a9a7c65b1de75818 in term 1.
I20260619 08:19:13.043318 9476 raft_consensus.cc:2454] T ffdef6ab9c1940c49f891bd0ac685225 P 8a95e1aebd794fb2ac2e3b0c705090e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd6378f8fa624434a9a7c65b1de75818 in term 1.
I20260619 08:19:13.043512 9543 leader_election.cc:302] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74beda5e6bdb4336a99c7df448188859, bd6378f8fa624434a9a7c65b1de75818; no voters:
I20260619 08:19:13.043591 9805 raft_consensus.cc:2781] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:13.043753 9805 raft_consensus.cc:686] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818 [term 1 LEADER]: Becoming Leader. State: Replica: bd6378f8fa624434a9a7c65b1de75818, State: Running, Role: LEADER
I20260619 08:19:13.043881 9805 consensus_queue.cc:237] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74beda5e6bdb4336a99c7df448188859" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41079 } } peers { permanent_uuid: "bd6378f8fa624434a9a7c65b1de75818" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37227 } } peers { permanent_uuid: "8a95e1aebd794fb2ac2e3b0c705090e7" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 36125 } }
I20260619 08:19:13.044759 9333 catalog_manager.cc:5495] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd6378f8fa624434a9a7c65b1de75818 (127.4.151.130). New cstate: current_term: 1 leader_uuid: "bd6378f8fa624434a9a7c65b1de75818" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74beda5e6bdb4336a99c7df448188859" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd6378f8fa624434a9a7c65b1de75818" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a95e1aebd794fb2ac2e3b0c705090e7" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 36125 } health_report { overall_health: UNKNOWN } } }
I20260619 08:19:13.083962 9796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:13.082742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49258 (local address 127.4.151.130:37227)
0619 08:19:13.082811 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:13.082813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:13.082825 (+ 12us) server_negotiation.cc:407] Connection header received
0619 08:19:13.082870 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:13.082871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:13.082892 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:13.083031 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:13.083688 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:13.083690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:13.083690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:19:13.083695 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:13.083718 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:19:13.083722 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:13.083784 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:13.083863 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 08:19:13.083877 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 08:19:13.086062 9738 raft_consensus.cc:1261] T ffdef6ab9c1940c49f891bd0ac685225 P 74beda5e6bdb4336a99c7df448188859 [term 1 FOLLOWER]: Refusing update from remote peer bd6378f8fa624434a9a7c65b1de75818: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:19:13.086199 9476 raft_consensus.cc:1261] T ffdef6ab9c1940c49f891bd0ac685225 P 8a95e1aebd794fb2ac2e3b0c705090e7 [term 1 FOLLOWER]: Refusing update from remote peer bd6378f8fa624434a9a7c65b1de75818: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:19:13.086488 9808 consensus_queue.cc:1035] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74beda5e6bdb4336a99c7df448188859" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:19:13.086639 9805 consensus_queue.cc:1035] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a95e1aebd794fb2ac2e3b0c705090e7" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:19:13.092600 9333 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:19:13.092727 9333 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:19:13.093642 9333 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d25679ead9c847a0a87d48b31fcf3417: Sending DeleteTablet for 3 replicas of tablet ffdef6ab9c1940c49f891bd0ac685225
I20260619 08:19:13.094051 9718 tablet_service.cc:1514] Processing DeleteTablet for tablet ffdef6ab9c1940c49f891bd0ac685225 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:13 UTC) from {username='slave'} at 127.0.0.1:49164
I20260619 08:19:13.094169 9587 tablet_service.cc:1514] Processing DeleteTablet for tablet ffdef6ab9c1940c49f891bd0ac685225 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:13 UTC) from {username='slave'} at 127.0.0.1:49252
I20260619 08:19:13.094142 9456 tablet_service.cc:1514] Processing DeleteTablet for tablet ffdef6ab9c1940c49f891bd0ac685225 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:13 UTC) from {username='slave'} at 127.0.0.1:59424
I20260619 08:19:13.094296 9822 tablet_replica.cc:330] T ffdef6ab9c1940c49f891bd0ac685225 P 74beda5e6bdb4336a99c7df448188859: stopping tablet replica
I20260619 08:19:13.094390 9822 raft_consensus.cc:2229] T ffdef6ab9c1940c49f891bd0ac685225 P 74beda5e6bdb4336a99c7df448188859 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:19:13.094607 9822 raft_consensus.cc:2258] T ffdef6ab9c1940c49f891bd0ac685225 P 74beda5e6bdb4336a99c7df448188859 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:13.094657 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 9394
I20260619 08:19:13.097533 9824 tablet_replica.cc:330] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818: stopping tablet replica
I20260619 08:19:13.097692 9824 raft_consensus.cc:2229] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:19:13.097759 9822 ts_tablet_manager.cc:1905] T ffdef6ab9c1940c49f891bd0ac685225 P 74beda5e6bdb4336a99c7df448188859: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:13.097831 9824 raft_consensus.cc:2258] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:13.098338 9824 ts_tablet_manager.cc:1905] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:13.099931 9822 ts_tablet_manager.cc:1918] T ffdef6ab9c1940c49f891bd0ac685225 P 74beda5e6bdb4336a99c7df448188859: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:19:13.100016 9822 log.cc:1196] T ffdef6ab9c1940c49f891bd0ac685225 P 74beda5e6bdb4336a99c7df448188859: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/ffdef6ab9c1940c49f891bd0ac685225
I20260619 08:19:13.100294 9822 ts_tablet_manager.cc:1939] T ffdef6ab9c1940c49f891bd0ac685225 P 74beda5e6bdb4336a99c7df448188859: Deleting consensus metadata
I20260619 08:19:13.100751 9321 catalog_manager.cc:4841] TS 74beda5e6bdb4336a99c7df448188859 (127.4.151.131:41079): tablet ffdef6ab9c1940c49f891bd0ac685225 (table test-table [id=ab79f077fb2f45c7a18c28a44102333b]) successfully deleted
I20260619 08:19:13.100811 9824 ts_tablet_manager.cc:1918] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:19:13.101074 9824 log.cc:1196] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/ffdef6ab9c1940c49f891bd0ac685225
I20260619 08:19:13.101583 9824 ts_tablet_manager.cc:1939] T ffdef6ab9c1940c49f891bd0ac685225 P bd6378f8fa624434a9a7c65b1de75818: Deleting consensus metadata
I20260619 08:19:13.102043 9318 catalog_manager.cc:4841] TS bd6378f8fa624434a9a7c65b1de75818 (127.4.151.130:37227): tablet ffdef6ab9c1940c49f891bd0ac685225 (table test-table [id=ab79f077fb2f45c7a18c28a44102333b]) successfully deleted
W20260619 08:19:13.102065 9321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.151.129:36125 (error 108)
I20260619 08:19:13.102209 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 9525
I20260619 08:19:13.103252 9792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:13.103104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:36125 (local address 127.0.0.1:59438)
0619 08:19:13.103172 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:13.103223 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.129:36125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 08:19:13.103375 9321 catalog_manager.cc:4584] TS 8a95e1aebd794fb2ac2e3b0c705090e7 (127.4.151.129:36125): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ffdef6ab9c1940c49f891bd0ac685225: Network error: Client connection negotiation failed: client connection to 127.4.151.129:36125: connect: Connection refused (error 111)
I20260619 08:19:13.107457 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 9656
I20260619 08:19:13.112033 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 9303
2026-06-19T08:19:13Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (751 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 08:19:13.131769 4702 mini_oidc.cc:152] Starting JWKS server
I20260619 08:19:13.132838 4702 webserver.cc:460] Webserver started at https://127.0.0.1:45145/ using document root <none> and password file <none>
I20260619 08:19:13.132965 4702 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 08:19:13.133180 4702 webserver.cc:460] Webserver started at http://127.0.0.1:41539/ using document root <none> and password file <none>
2026-06-19T08:19:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:19:13Z Disabled control of system clock
I20260619 08:19:13.141232 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:45005
--webserver_interface=127.4.151.190
--webserver_port=0
--builtin_ntp_servers=127.4.151.148:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:45005
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45145/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781857130600961-4702-0/testchainca.cert with env {}
W20260619 08:19:13.224546 9831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:13.224773 9831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:13.224795 9831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:13.226073 9831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:19:13.226121 9831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:13.226133 9831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:13.226146 9831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:19:13.226159 9831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 08:19:13.226171 9831 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 08:19:13.226182 9831 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45145/jwks.json
I20260619 08:19:13.227699 9831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:45005
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:45005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45145/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781857130600961-4702-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:13.227895 9831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:13.228062 9831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:13.233222 9839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:13.233299 9831 server_base.cc:1034] running on GCE node
W20260619 08:19:13.233222 9837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:13.233575 9836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:13.233826 9831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:13.234167 9831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:13.235323 9831 hybrid_clock.cc:648] HybridClock initialized: now 1781857153235300 us; error 53 us; skew 500 ppm
I20260619 08:19:13.236730 9831 webserver.cc:460] Webserver started at http://127.4.151.190:43299/ using document root <none> and password file <none>
I20260619 08:19:13.236943 9831 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:13.236985 9831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:13.237087 9831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:13.238189 9831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "ce13635132814f2bbaccc0c8e3fe3c82"
format_stamp: "Formatted at 2026-06-19 08:19:13 on dist-test-slave-d3n8"
server_key: "2d61f9c2e75cefa1f6a791d368bac4d4"
server_key_iv: "dca99e8f17555cd199c46925c72d5b29"
server_key_version: "encryptionkey@0"
I20260619 08:19:13.238492 9831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "ce13635132814f2bbaccc0c8e3fe3c82"
format_stamp: "Formatted at 2026-06-19 08:19:13 on dist-test-slave-d3n8"
server_key: "2d61f9c2e75cefa1f6a791d368bac4d4"
server_key_iv: "dca99e8f17555cd199c46925c72d5b29"
server_key_version: "encryptionkey@0"
I20260619 08:19:13.239799 9831 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 08:19:13.240717 9845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:13.240959 9831 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:13.241055 9831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "ce13635132814f2bbaccc0c8e3fe3c82"
format_stamp: "Formatted at 2026-06-19 08:19:13 on dist-test-slave-d3n8"
server_key: "2d61f9c2e75cefa1f6a791d368bac4d4"
server_key_iv: "dca99e8f17555cd199c46925c72d5b29"
server_key_version: "encryptionkey@0"
I20260619 08:19:13.241182 9831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:13.259414 9831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:13.259763 9831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:13.259922 9831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:13.263850 9831 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:45005
I20260619 08:19:13.263882 9897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:45005 every 8 connection(s)
I20260619 08:19:13.264274 9831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:19:13.265247 9898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:13.265801 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 9831
I20260619 08:19:13.265919 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:19:13.266187 4702 external_mini_cluster.cc:1428] Setting key 074bd3e8cd76c58bdc8dbbf94290eefe
I20260619 08:19:13.268448 9898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce13635132814f2bbaccc0c8e3fe3c82: Bootstrap starting.
I20260619 08:19:13.269155 9898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce13635132814f2bbaccc0c8e3fe3c82: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:13.269475 9898 log.cc:824] T 00000000000000000000000000000000 P ce13635132814f2bbaccc0c8e3fe3c82: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:13.270197 9898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce13635132814f2bbaccc0c8e3fe3c82: No bootstrap required, opened a new log
I20260619 08:19:13.272411 9901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:13.266758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45094 (local address 127.4.151.190:45005)
0619 08:19:13.266972 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:13.266980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:13.267012 (+ 32us) server_negotiation.cc:407] Connection header received
0619 08:19:13.267589 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:13.267598 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:13.267679 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:13.267826 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:13.268014 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:13.269348 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:13.270172 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:13.270769 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:13.270895 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:13.271547 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:13.271551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:13.271551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:19:13.271763 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:13.271812 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:19:13.271829 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:13.271911 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:13.272092 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 08:19:13.272200 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
I20260619 08:19:13.272459 9898 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce13635132814f2bbaccc0c8e3fe3c82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce13635132814f2bbaccc0c8e3fe3c82" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 45005 } }
I20260619 08:19:13.272606 9898 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce13635132814f2bbaccc0c8e3fe3c82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:13.272647 9898 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce13635132814f2bbaccc0c8e3fe3c82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce13635132814f2bbaccc0c8e3fe3c82, State: Initialized, Role: FOLLOWER
I20260619 08:19:13.272765 9898 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce13635132814f2bbaccc0c8e3fe3c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce13635132814f2bbaccc0c8e3fe3c82" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 45005 } }
I20260619 08:19:13.272841 9898 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce13635132814f2bbaccc0c8e3fe3c82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:19:13.272881 9898 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce13635132814f2bbaccc0c8e3fe3c82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:19:13.272938 9898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce13635132814f2bbaccc0c8e3fe3c82 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:13.273474 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:19:13.273663 9898 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce13635132814f2bbaccc0c8e3fe3c82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce13635132814f2bbaccc0c8e3fe3c82" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 45005 } }
I20260619 08:19:13.273759 9898 leader_election.cc:302] T 00000000000000000000000000000000 P ce13635132814f2bbaccc0c8e3fe3c82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce13635132814f2bbaccc0c8e3fe3c82; no voters:
I20260619 08:19:13.273927 9898 leader_election.cc:288] T 00000000000000000000000000000000 P ce13635132814f2bbaccc0c8e3fe3c82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:19:13.273981 9903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce13635132814f2bbaccc0c8e3fe3c82 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:13.274133 9903 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce13635132814f2bbaccc0c8e3fe3c82 [term 1 LEADER]: Becoming Leader. State: Replica: ce13635132814f2bbaccc0c8e3fe3c82, State: Running, Role: LEADER
I20260619 08:19:13.274194 9898 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce13635132814f2bbaccc0c8e3fe3c82 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:19:13.274228 9903 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce13635132814f2bbaccc0c8e3fe3c82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce13635132814f2bbaccc0c8e3fe3c82" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 45005 } }
I20260619 08:19:13.274521 9903 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce13635132814f2bbaccc0c8e3fe3c82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce13635132814f2bbaccc0c8e3fe3c82. Latest consensus state: current_term: 1 leader_uuid: "ce13635132814f2bbaccc0c8e3fe3c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce13635132814f2bbaccc0c8e3fe3c82" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 45005 } } }
I20260619 08:19:13.274613 9903 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce13635132814f2bbaccc0c8e3fe3c82 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:13.274531 9905 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce13635132814f2bbaccc0c8e3fe3c82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce13635132814f2bbaccc0c8e3fe3c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce13635132814f2bbaccc0c8e3fe3c82" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 45005 } } }
I20260619 08:19:13.274710 9905 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce13635132814f2bbaccc0c8e3fe3c82 [sys.catalog]: This master's current role is: LEADER
W20260619 08:19:13.275812 9918 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce13635132814f2bbaccc0c8e3fe3c82: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:19:13.275877 9918 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:19:13.275983 9909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:19:13.276153 9909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:19:13.277648 9909 catalog_manager.cc:1347] Generated new cluster ID: 049b5f4b789840248af56d445f6df9f0
I20260619 08:19:13.277714 9909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:19:13.301515 9909 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:19:13.302060 9909 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:19:13.307025 9909 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce13635132814f2bbaccc0c8e3fe3c82: Generated new TSK 0
I20260619 08:19:13.307237 9909 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 08:19:13.344416 9928 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 08:19:13.344561 9928 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 08:19:13.344738 9901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:13.321896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45106 (local address 127.4.151.190:45005)
0619 08:19:13.321971 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:13.321973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:13.322103 (+ 130us) server_negotiation.cc:407] Connection header received
0619 08:19:13.322205 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:13.322206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:13.322217 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:13.322278 (+ 61us) server_negotiation.cc:226] Negotiated authn=JWT
0619 08:19:13.322443 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:13.323014 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:13.324192 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:13.324394 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:13.324463 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:13.324705 (+ 242us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 08:19:13.344428 (+ 19723us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 08:19:13.344639 (+ 211us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45106: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 08:19:13.344792 9901 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45106: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 08:19:13.345119 9927 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:13.321833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:45005 (local address 127.0.0.1:45106)
0619 08:19:13.322024 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:13.322036 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:13.322109 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:13.322301 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:13.322302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:13.322303 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0619 08:19:13.322377 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:13.322381 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:13.323115 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:13.323116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:13.324090 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:13.324093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:13.324478 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:13.324479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:13.324546 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:13.324572 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 08:19:13.344590 (+ 20018us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 08:19:13.344733 (+ 143us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.151.190:45005: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":105,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.4.151.190:45005: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 08:19:13.345621 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 9831
2026-06-19T08:19:13Z chronyd exiting
I20260619 08:19:13.382321 4702 test_util.cc:175] -----------------------------------------------
I20260619 08:19:13.382416 4702 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781857130600961-4702-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (251 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 08:19:13.383387 4702 mini_oidc.cc:152] Starting JWKS server
I20260619 08:19:13.384052 4702 webserver.cc:460] Webserver started at https://127.0.0.1:36453/ using document root <none> and password file <none>
I20260619 08:19:13.384092 4702 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 08:19:13.384249 4702 webserver.cc:460] Webserver started at http://127.0.0.1:45795/ using document root <none> and password file <none>
2026-06-19T08:19:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:19:13Z Disabled control of system clock
I20260619 08:19:13.392226 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:33945
--webserver_interface=127.4.151.190
--webserver_port=0
--builtin_ntp_servers=127.4.151.148:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:33945
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36453/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781857130600961-4702-0/testchainca.cert with env {}
W20260619 08:19:13.476814 9934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:13.477028 9934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:13.477051 9934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:13.478214 9934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:19:13.478252 9934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:13.478264 9934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:13.478277 9934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:19:13.478291 9934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 08:19:13.478302 9934 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 08:19:13.478313 9934 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36453/jwks.json
I20260619 08:19:13.479735 9934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:33945
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:33945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36453/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781857130600961-4702-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:13.479940 9934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:13.480123 9934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:13.485637 9942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:13.485857 9934 server_base.cc:1034] running on GCE node
W20260619 08:19:13.485690 9939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:13.485662 9940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:13.486328 9934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:13.486625 9934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:13.487802 9934 hybrid_clock.cc:648] HybridClock initialized: now 1781857153487787 us; error 51 us; skew 500 ppm
I20260619 08:19:13.489064 9934 webserver.cc:460] Webserver started at http://127.4.151.190:41789/ using document root <none> and password file <none>
I20260619 08:19:13.489320 9934 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:13.489408 9934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:13.489571 9934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:13.490680 9934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "1b37d14131ef479ba72d345628149579"
format_stamp: "Formatted at 2026-06-19 08:19:13 on dist-test-slave-d3n8"
server_key: "c3ed5e44a75b136378c90bac1dcebef6"
server_key_iv: "0bf82864e32d71bee27ad2bdd30dccd8"
server_key_version: "encryptionkey@0"
I20260619 08:19:13.491062 9934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "1b37d14131ef479ba72d345628149579"
format_stamp: "Formatted at 2026-06-19 08:19:13 on dist-test-slave-d3n8"
server_key: "c3ed5e44a75b136378c90bac1dcebef6"
server_key_iv: "0bf82864e32d71bee27ad2bdd30dccd8"
server_key_version: "encryptionkey@0"
I20260619 08:19:13.492533 9934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:19:13.493484 9948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:13.493685 9934 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:13.493793 9934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "1b37d14131ef479ba72d345628149579"
format_stamp: "Formatted at 2026-06-19 08:19:13 on dist-test-slave-d3n8"
server_key: "c3ed5e44a75b136378c90bac1dcebef6"
server_key_iv: "0bf82864e32d71bee27ad2bdd30dccd8"
server_key_version: "encryptionkey@0"
I20260619 08:19:13.493892 9934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:13.507768 9934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:13.508141 9934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:13.508296 9934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:13.512367 9934 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:33945
I20260619 08:19:13.512400 10000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:33945 every 8 connection(s)
I20260619 08:19:13.512741 9934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:19:13.513818 10001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:13.516230 10001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b37d14131ef479ba72d345628149579: Bootstrap starting.
I20260619 08:19:13.516611 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 9934
I20260619 08:19:13.516738 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:19:13.516899 10001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b37d14131ef479ba72d345628149579: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:13.516927 4702 external_mini_cluster.cc:1428] Setting key e9c7746e8d71394952e3218637e494dc
I20260619 08:19:13.517195 10001 log.cc:824] T 00000000000000000000000000000000 P 1b37d14131ef479ba72d345628149579: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:13.517956 10001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b37d14131ef479ba72d345628149579: No bootstrap required, opened a new log
I20260619 08:19:13.520113 10001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b37d14131ef479ba72d345628149579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b37d14131ef479ba72d345628149579" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33945 } }
I20260619 08:19:13.520282 10001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b37d14131ef479ba72d345628149579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:13.520313 10001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b37d14131ef479ba72d345628149579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b37d14131ef479ba72d345628149579, State: Initialized, Role: FOLLOWER
I20260619 08:19:13.520434 10001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b37d14131ef479ba72d345628149579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b37d14131ef479ba72d345628149579" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33945 } }
I20260619 08:19:13.520522 10001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b37d14131ef479ba72d345628149579 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:19:13.520552 10001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b37d14131ef479ba72d345628149579 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:19:13.520599 10001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b37d14131ef479ba72d345628149579 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:13.521143 10001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b37d14131ef479ba72d345628149579 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b37d14131ef479ba72d345628149579" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33945 } }
I20260619 08:19:13.521265 10001 leader_election.cc:302] T 00000000000000000000000000000000 P 1b37d14131ef479ba72d345628149579 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b37d14131ef479ba72d345628149579; no voters:
I20260619 08:19:13.521488 10001 leader_election.cc:288] T 00000000000000000000000000000000 P 1b37d14131ef479ba72d345628149579 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:19:13.521520 10006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b37d14131ef479ba72d345628149579 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:13.521656 10006 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b37d14131ef479ba72d345628149579 [term 1 LEADER]: Becoming Leader. State: Replica: 1b37d14131ef479ba72d345628149579, State: Running, Role: LEADER
I20260619 08:19:13.521739 10006 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b37d14131ef479ba72d345628149579 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b37d14131ef479ba72d345628149579" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33945 } }
I20260619 08:19:13.522075 10007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b37d14131ef479ba72d345628149579 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b37d14131ef479ba72d345628149579. Latest consensus state: current_term: 1 leader_uuid: "1b37d14131ef479ba72d345628149579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b37d14131ef479ba72d345628149579" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33945 } } }
I20260619 08:19:13.522068 10006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b37d14131ef479ba72d345628149579 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b37d14131ef479ba72d345628149579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b37d14131ef479ba72d345628149579" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33945 } } }
I20260619 08:19:13.522161 10001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b37d14131ef479ba72d345628149579 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:19:13.522174 10007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b37d14131ef479ba72d345628149579 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:13.522178 10006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b37d14131ef479ba72d345628149579 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:13.522580 10010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:19:13.523051 10005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:13.517471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37258 (local address 127.4.151.190:33945)
0619 08:19:13.517673 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:13.517682 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:13.517719 (+ 37us) server_negotiation.cc:407] Connection header received
0619 08:19:13.518238 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:13.518245 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:13.518302 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:13.518411 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:13.518656 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:13.519529 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:13.520674 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:13.521163 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:13.521317 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:13.522102 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:13.522106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:13.522106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:19:13.522318 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:13.522367 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:19:13.522381 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:13.522444 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:13.522716 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 08:19:13.522812 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260619 08:19:13.523306 10010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:19:13.524004 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:19:13.524966 10010 catalog_manager.cc:1347] Generated new cluster ID: a7a6edc36e044a7f9bbed1d6d6418b4d
I20260619 08:19:13.525019 10010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:19:13.563699 10010 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:19:13.564426 10010 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:19:13.570647 10010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b37d14131ef479ba72d345628149579: Generated new TSK 0
I20260619 08:19:13.570835 10010 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 08:19:13.604842 10031 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 08:19:13.604939 10031 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 08:19:13.605260 10005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:13.583368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37262 (local address 127.4.151.190:33945)
0619 08:19:13.583421 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:13.583423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:13.583523 (+ 100us) server_negotiation.cc:407] Connection header received
0619 08:19:13.583596 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:13.583597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:13.583608 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:13.583675 (+ 67us) server_negotiation.cc:226] Negotiated authn=JWT
0619 08:19:13.583827 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:13.584436 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:13.585667 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:13.585879 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:13.585934 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:13.586126 (+ 192us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 08:19:13.604965 (+ 18839us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36453/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 08:19:13.605175 (+ 210us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37262: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36453/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":27}
W20260619 08:19:13.605360 10005 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37262: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36453/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 08:19:13.605400 10030 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:13.583268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33945 (local address 127.0.0.1:37262)
0619 08:19:13.583437 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:13.583450 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:13.583523 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:13.583691 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:13.583692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:13.583694 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 08:19:13.583757 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:13.583759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:13.584578 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:13.584579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:13.585570 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:13.585573 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:13.585960 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:13.585961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:13.586021 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:13.586027 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 08:19:13.605157 (+ 19130us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36453/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 08:19:13.605281 (+ 124us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.151.190:33945: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36453/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260619 08:19:13.605885 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 9934
2026-06-19T08:19:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (260 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 08:19:13.643611 4702 mini_oidc.cc:152] Starting JWKS server
I20260619 08:19:13.644265 4702 webserver.cc:460] Webserver started at https://127.0.0.1:45515/ using document root <none> and password file <none>
I20260619 08:19:13.644305 4702 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 08:19:13.644454 4702 webserver.cc:460] Webserver started at http://127.0.0.1:41039/ using document root <none> and password file <none>
2026-06-19T08:19:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:19:13Z Disabled control of system clock
I20260619 08:19:13.652415 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:33773
--webserver_interface=127.4.151.190
--webserver_port=0
--builtin_ntp_servers=127.4.151.148:33723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:33773
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45515/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 08:19:13.734218 10037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:13.734485 10037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:13.734524 10037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:13.735721 10037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:19:13.735796 10037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:13.735818 10037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:13.735841 10037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:19:13.735869 10037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 08:19:13.735893 10037 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 08:19:13.735922 10037 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45515/jwks.json
I20260619 08:19:13.737322 10037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:33723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:33773
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:33773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45515/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:13.737687 10037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:13.738011 10037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:13.743372 10043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:13.743397 10045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:13.743367 10042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:13.743642 10037 server_base.cc:1034] running on GCE node
I20260619 08:19:13.743934 10037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:13.744194 10037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:13.745333 10037 hybrid_clock.cc:648] HybridClock initialized: now 1781857153745314 us; error 37 us; skew 500 ppm
I20260619 08:19:13.746701 10037 webserver.cc:460] Webserver started at http://127.4.151.190:37009/ using document root <none> and password file <none>
I20260619 08:19:13.746953 10037 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:13.746999 10037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:13.747139 10037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:13.748234 10037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "eb6dfc9ab7c2427d8314adfe5552159f"
format_stamp: "Formatted at 2026-06-19 08:19:13 on dist-test-slave-d3n8"
server_key: "f76383d177003971a37ed487afc7d601"
server_key_iv: "93f643ab38e4962d57b392495bd610d1"
server_key_version: "encryptionkey@0"
I20260619 08:19:13.748596 10037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "eb6dfc9ab7c2427d8314adfe5552159f"
format_stamp: "Formatted at 2026-06-19 08:19:13 on dist-test-slave-d3n8"
server_key: "f76383d177003971a37ed487afc7d601"
server_key_iv: "93f643ab38e4962d57b392495bd610d1"
server_key_version: "encryptionkey@0"
I20260619 08:19:13.750049 10037 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:19:13.750873 10051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:13.751231 10037 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:13.751354 10037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "eb6dfc9ab7c2427d8314adfe5552159f"
format_stamp: "Formatted at 2026-06-19 08:19:13 on dist-test-slave-d3n8"
server_key: "f76383d177003971a37ed487afc7d601"
server_key_iv: "93f643ab38e4962d57b392495bd610d1"
server_key_version: "encryptionkey@0"
I20260619 08:19:13.751435 10037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:13.789654 10037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:13.790000 10037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:13.790165 10037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:13.794608 10037 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:33773
I20260619 08:19:13.794651 10103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:33773 every 8 connection(s)
I20260619 08:19:13.795030 10037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:19:13.795742 10104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:13.797545 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 10037
I20260619 08:19:13.797663 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:19:13.797856 4702 external_mini_cluster.cc:1428] Setting key dd49a9fb5d2a135b8954fead85edfc2b
I20260619 08:19:13.799620 10104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6dfc9ab7c2427d8314adfe5552159f: Bootstrap starting.
I20260619 08:19:13.800431 10104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb6dfc9ab7c2427d8314adfe5552159f: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:13.800776 10104 log.cc:824] T 00000000000000000000000000000000 P eb6dfc9ab7c2427d8314adfe5552159f: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:13.801651 10104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6dfc9ab7c2427d8314adfe5552159f: No bootstrap required, opened a new log
I20260619 08:19:13.804432 10104 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb6dfc9ab7c2427d8314adfe5552159f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6dfc9ab7c2427d8314adfe5552159f" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33773 } }
I20260619 08:19:13.804620 10104 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb6dfc9ab7c2427d8314adfe5552159f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:13.804659 10104 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb6dfc9ab7c2427d8314adfe5552159f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6dfc9ab7c2427d8314adfe5552159f, State: Initialized, Role: FOLLOWER
I20260619 08:19:13.804695 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:13.798427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48570 (local address 127.4.151.190:33773)
0619 08:19:13.798689 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:13.798701 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:13.798737 (+ 36us) server_negotiation.cc:407] Connection header received
0619 08:19:13.799381 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:13.799390 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:13.799489 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:13.799659 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:13.799925 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:13.801432 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:13.802315 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:13.803006 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:13.803153 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:13.803910 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:13.803916 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:13.803917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:19:13.804173 (+ 256us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:13.804220 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:19:13.804236 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:13.804313 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:13.804379 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 08:19:13.804477 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":82,"threads_started":1}
I20260619 08:19:13.805040 10104 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb6dfc9ab7c2427d8314adfe5552159f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6dfc9ab7c2427d8314adfe5552159f" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33773 } }
I20260619 08:19:13.805146 10104 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb6dfc9ab7c2427d8314adfe5552159f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:19:13.805186 10104 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb6dfc9ab7c2427d8314adfe5552159f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:19:13.805229 10104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb6dfc9ab7c2427d8314adfe5552159f [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:13.805855 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:19:13.806035 10104 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb6dfc9ab7c2427d8314adfe5552159f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6dfc9ab7c2427d8314adfe5552159f" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33773 } }
I20260619 08:19:13.806162 10104 leader_election.cc:302] T 00000000000000000000000000000000 P eb6dfc9ab7c2427d8314adfe5552159f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb6dfc9ab7c2427d8314adfe5552159f; no voters:
I20260619 08:19:13.806378 10104 leader_election.cc:288] T 00000000000000000000000000000000 P eb6dfc9ab7c2427d8314adfe5552159f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:19:13.806403 10109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb6dfc9ab7c2427d8314adfe5552159f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:13.806526 10109 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb6dfc9ab7c2427d8314adfe5552159f [term 1 LEADER]: Becoming Leader. State: Replica: eb6dfc9ab7c2427d8314adfe5552159f, State: Running, Role: LEADER
I20260619 08:19:13.806614 10109 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb6dfc9ab7c2427d8314adfe5552159f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6dfc9ab7c2427d8314adfe5552159f" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33773 } }
I20260619 08:19:13.806658 10104 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb6dfc9ab7c2427d8314adfe5552159f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:19:13.806967 10111 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6dfc9ab7c2427d8314adfe5552159f [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb6dfc9ab7c2427d8314adfe5552159f. Latest consensus state: current_term: 1 leader_uuid: "eb6dfc9ab7c2427d8314adfe5552159f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6dfc9ab7c2427d8314adfe5552159f" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33773 } } }
I20260619 08:19:13.807067 10111 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6dfc9ab7c2427d8314adfe5552159f [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:13.807248 10110 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6dfc9ab7c2427d8314adfe5552159f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb6dfc9ab7c2427d8314adfe5552159f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6dfc9ab7c2427d8314adfe5552159f" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33773 } } }
I20260619 08:19:13.807307 10110 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6dfc9ab7c2427d8314adfe5552159f [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:13.807322 10114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:19:13.807938 10114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:19:13.809640 10114 catalog_manager.cc:1347] Generated new cluster ID: f87b69ec31374c0dbe384098888a708e
I20260619 08:19:13.809697 10114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:19:13.828008 10114 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:19:13.828733 10114 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:19:13.831023 10114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb6dfc9ab7c2427d8314adfe5552159f: Generated new TSK 0
I20260619 08:19:13.831211 10114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:19:13.845647 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:13.841637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48576 (local address 127.4.151.190:33773)
0619 08:19:13.841700 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:13.841703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:13.841850 (+ 147us) server_negotiation.cc:407] Connection header received
0619 08:19:13.841909 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:13.841910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:13.841922 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:13.842013 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0619 08:19:13.842118 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:13.842928 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:13.843752 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:13.844079 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:13.844152 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:13.844367 (+ 215us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 08:19:13.845428 (+ 1061us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45515/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 08:19:13.845561 (+ 133us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48576: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45515/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 08:19:13.845712 10107 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48576: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45515/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 08:19:13.845743 10133 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:13.841546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33773 (local address 127.0.0.1:48576)
0619 08:19:13.841781 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:13.841792 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:13.841848 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:13.842010 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:13.842010 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:13.842012 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 08:19:13.842062 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:13.842064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:13.843014 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:13.843015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:13.843664 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:13.843667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:13.844186 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:13.844187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:13.844254 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:13.844260 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 08:19:13.845531 (+ 1271us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45515/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 08:19:13.845641 (+ 110us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.151.190:33773: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45515/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":125,"threads_started":1}
I20260619 08:19:13.846402 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 10037
E20260619 08:19:13.856801 10134 webserver.cc:533] Webserver: sslize failed:
E20260619 08:19:13.856882 10134 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T08:19:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (240 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T08:19:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:19:13Z Disabled control of system clock
I20260619 08:19:13.892172 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:40245
--webserver_interface=127.4.151.190
--webserver_port=0
--builtin_ntp_servers=127.4.151.148:34563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:40245
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781857130600961-4702-0/testchainca.cert with env {}
W20260619 08:19:13.973453 10138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:13.973672 10138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:13.973695 10138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:13.974849 10138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:19:13.974893 10138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:13.974906 10138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:13.974918 10138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:19:13.974932 10138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 08:19:13.974944 10138 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 08:19:13.974956 10138 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 08:19:13.976414 10138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:34563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:40245
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:40245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781857130600961-4702-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:13.976611 10138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:13.976779 10138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:13.982209 10143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:13.982220 10144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:13.982240 10138 server_base.cc:1034] running on GCE node
W20260619 08:19:13.982209 10146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:13.982684 10138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:13.982944 10138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:13.984161 10138 hybrid_clock.cc:648] HybridClock initialized: now 1781857153984098 us; error 53 us; skew 500 ppm
I20260619 08:19:13.985736 10138 webserver.cc:460] Webserver started at http://127.4.151.190:35311/ using document root <none> and password file <none>
I20260619 08:19:13.986022 10138 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:13.986086 10138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:13.986238 10138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:13.987326 10138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "0a29c270ab5144b2b4035cd8e4257516"
format_stamp: "Formatted at 2026-06-19 08:19:13 on dist-test-slave-d3n8"
server_key: "663639a1c6dd17ce3c095a887cdef532"
server_key_iv: "936d51afdd49172ad383d846d2c1cddf"
server_key_version: "encryptionkey@0"
I20260619 08:19:13.987639 10138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "0a29c270ab5144b2b4035cd8e4257516"
format_stamp: "Formatted at 2026-06-19 08:19:13 on dist-test-slave-d3n8"
server_key: "663639a1c6dd17ce3c095a887cdef532"
server_key_iv: "936d51afdd49172ad383d846d2c1cddf"
server_key_version: "encryptionkey@0"
I20260619 08:19:13.988945 10138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:19:13.989715 10152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:13.989892 10138 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:13.990078 10138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "0a29c270ab5144b2b4035cd8e4257516"
format_stamp: "Formatted at 2026-06-19 08:19:13 on dist-test-slave-d3n8"
server_key: "663639a1c6dd17ce3c095a887cdef532"
server_key_iv: "936d51afdd49172ad383d846d2c1cddf"
server_key_version: "encryptionkey@0"
I20260619 08:19:13.990160 10138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:14.020664 10138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:14.021071 10138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:14.021240 10138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:14.025048 10138 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:40245
I20260619 08:19:14.025130 10204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:40245 every 8 connection(s)
I20260619 08:19:14.025514 10138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:19:14.026535 10205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:14.026980 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 10138
I20260619 08:19:14.027099 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:19:14.027294 4702 external_mini_cluster.cc:1428] Setting key 4c1c138becf73de4162370a256f4df18
I20260619 08:19:14.029924 10205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a29c270ab5144b2b4035cd8e4257516: Bootstrap starting.
I20260619 08:19:14.030642 10205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a29c270ab5144b2b4035cd8e4257516: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:14.030978 10205 log.cc:824] T 00000000000000000000000000000000 P 0a29c270ab5144b2b4035cd8e4257516: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:14.031739 10205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a29c270ab5144b2b4035cd8e4257516: No bootstrap required, opened a new log
I20260619 08:19:14.033888 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:14.027846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43500 (local address 127.4.151.190:40245)
0619 08:19:14.028075 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:14.028086 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:14.028123 (+ 37us) server_negotiation.cc:407] Connection header received
0619 08:19:14.028739 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:14.028748 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:14.028827 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:14.028995 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:14.029171 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:14.030543 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:14.031508 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:14.032118 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:14.032300 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:14.033038 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:14.033042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:14.033042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:19:14.033271 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:14.033313 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:19:14.033327 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:14.033437 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:14.033533 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 08:19:14.033616 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":82,"threads_started":1}
I20260619 08:19:14.034422 10205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a29c270ab5144b2b4035cd8e4257516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a29c270ab5144b2b4035cd8e4257516" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 40245 } }
I20260619 08:19:14.034605 10205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a29c270ab5144b2b4035cd8e4257516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:14.034642 10205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a29c270ab5144b2b4035cd8e4257516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29c270ab5144b2b4035cd8e4257516, State: Initialized, Role: FOLLOWER
I20260619 08:19:14.034781 10205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a29c270ab5144b2b4035cd8e4257516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a29c270ab5144b2b4035cd8e4257516" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 40245 } }
I20260619 08:19:14.034873 10205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a29c270ab5144b2b4035cd8e4257516 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:19:14.034904 10205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a29c270ab5144b2b4035cd8e4257516 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:19:14.034955 10205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a29c270ab5144b2b4035cd8e4257516 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:14.035164 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:19:14.035712 10205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a29c270ab5144b2b4035cd8e4257516 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a29c270ab5144b2b4035cd8e4257516" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 40245 } }
I20260619 08:19:14.035830 10205 leader_election.cc:302] T 00000000000000000000000000000000 P 0a29c270ab5144b2b4035cd8e4257516 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a29c270ab5144b2b4035cd8e4257516; no voters:
I20260619 08:19:14.035998 10205 leader_election.cc:288] T 00000000000000000000000000000000 P 0a29c270ab5144b2b4035cd8e4257516 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:19:14.036055 10210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a29c270ab5144b2b4035cd8e4257516 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:14.036230 10210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a29c270ab5144b2b4035cd8e4257516 [term 1 LEADER]: Becoming Leader. State: Replica: 0a29c270ab5144b2b4035cd8e4257516, State: Running, Role: LEADER
I20260619 08:19:14.036253 10205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a29c270ab5144b2b4035cd8e4257516 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:19:14.036312 10210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a29c270ab5144b2b4035cd8e4257516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a29c270ab5144b2b4035cd8e4257516" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 40245 } }
I20260619 08:19:14.036748 10211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a29c270ab5144b2b4035cd8e4257516 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a29c270ab5144b2b4035cd8e4257516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a29c270ab5144b2b4035cd8e4257516" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 40245 } } }
I20260619 08:19:14.036859 10211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a29c270ab5144b2b4035cd8e4257516 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:14.037067 10212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a29c270ab5144b2b4035cd8e4257516 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a29c270ab5144b2b4035cd8e4257516. Latest consensus state: current_term: 1 leader_uuid: "0a29c270ab5144b2b4035cd8e4257516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a29c270ab5144b2b4035cd8e4257516" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 40245 } } }
I20260619 08:19:14.037122 10219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:19:14.037132 10212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a29c270ab5144b2b4035cd8e4257516 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:14.037895 10219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:19:14.039655 10219 catalog_manager.cc:1347] Generated new cluster ID: 2e8f9f79914643bd8e411ef3817b5d92
I20260619 08:19:14.039714 10219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:19:14.047008 10219 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:19:14.047580 10219 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:19:14.055332 10219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a29c270ab5144b2b4035cd8e4257516: Generated new TSK 0
I20260619 08:19:14.055557 10219 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:19:14.101187 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:14.063234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43508 (local address 127.4.151.190:40245)
0619 08:19:14.063305 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:14.063308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:14.063416 (+ 108us) server_negotiation.cc:407] Connection header received
0619 08:19:14.063499 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:14.063500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:14.063511 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:14.063577 (+ 66us) server_negotiation.cc:226] Negotiated authn=JWT
0619 08:19:14.063733 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:14.064335 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:14.065501 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:14.065709 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:14.065771 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:14.065958 (+ 187us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 08:19:14.100964 (+ 35006us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 08:19:14.101126 (+ 162us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43508: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 08:19:14.101289 10208 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43508: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 08:19:14.101361 10234 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:14.063174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40245 (local address 127.0.0.1:43508)
0619 08:19:14.063339 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:14.063352 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:14.063423 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:14.063599 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:14.063600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:14.063602 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 08:19:14.063666 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:14.063669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:14.064429 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:14.064430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:14.065423 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:14.065426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:14.065795 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:14.065796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:14.065859 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:14.065865 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 08:19:14.101117 (+ 35252us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 08:19:14.101261 (+ 144us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.151.190:40245: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":106,"threads_started":1}
I20260619 08:19:14.101886 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 10138
2026-06-19T08:19:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (234 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (89 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (89 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:19:14 dist-test-slave-d3n8 krb5kdc[10246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:19:14 dist-test-slave-d3n8 krb5kdc[10246](info): set up 2 sockets
Jun 19 08:19:14 dist-test-slave-d3n8 krb5kdc[10246](info): commencing operation
krb5kdc: starting...
W20260619 08:19:15.337637 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:19:15 dist-test-slave-d3n8 krb5kdc[10246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:19:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:19:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:19:15.464504 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:43297
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:43297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:15.546370 10259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:15.546654 10259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:15.546698 10259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:15.548089 10259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:19:15.548182 10259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:15.548205 10259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:15.548224 10259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:19:15.548256 10259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:19:15.549840 10259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:43297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:43297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:15.550137 10259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:15.550375 10259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:15.555528 10265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:15.555541 10264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:15.555860 10267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:15.556437 10259 server_base.cc:1034] running on GCE node
I20260619 08:19:15.556720 10259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:15.557034 10259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:15.558245 10259 hybrid_clock.cc:648] HybridClock initialized: now 1781857155558203 us; error 56 us; skew 500 ppm
Jun 19 08:19:15 dist-test-slave-d3n8 krb5kdc[10246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857155, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:15.560930 10259 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:19:15.561475 10259 webserver.cc:460] Webserver started at http://127.4.151.190:44421/ using document root <none> and password file <none>
I20260619 08:19:15.561664 10259 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:15.561707 10259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:15.561798 10259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:15.562853 10259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "1a69a4d7964d4220a080f04e1597a49c"
format_stamp: "Formatted at 2026-06-19 08:19:15 on dist-test-slave-d3n8"
server_key: "e22a1596df00e8f920c102a25a60835a"
server_key_iv: "16922d7987f2be0aa7255ba40076be46"
server_key_version: "encryptionkey@0"
I20260619 08:19:15.563192 10259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "1a69a4d7964d4220a080f04e1597a49c"
format_stamp: "Formatted at 2026-06-19 08:19:15 on dist-test-slave-d3n8"
server_key: "e22a1596df00e8f920c102a25a60835a"
server_key_iv: "16922d7987f2be0aa7255ba40076be46"
server_key_version: "encryptionkey@0"
I20260619 08:19:15.564494 10259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:19:15.565182 10274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:15.565442 10259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:15.565549 10259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "1a69a4d7964d4220a080f04e1597a49c"
format_stamp: "Formatted at 2026-06-19 08:19:15 on dist-test-slave-d3n8"
server_key: "e22a1596df00e8f920c102a25a60835a"
server_key_iv: "16922d7987f2be0aa7255ba40076be46"
server_key_version: "encryptionkey@0"
I20260619 08:19:15.565626 10259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:15.575515 10259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:15.577715 10259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:15.577867 10259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:15.581967 10259 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:43297
I20260619 08:19:15.582093 10326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:43297 every 8 connection(s)
I20260619 08:19:15.582398 10259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:19:15.583250 10327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:15.585743 10327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a69a4d7964d4220a080f04e1597a49c: Bootstrap starting.
I20260619 08:19:15.586360 10327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a69a4d7964d4220a080f04e1597a49c: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:15.586665 10327 log.cc:824] T 00000000000000000000000000000000 P 1a69a4d7964d4220a080f04e1597a49c: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:15.587518 10327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a69a4d7964d4220a080f04e1597a49c: No bootstrap required, opened a new log
I20260619 08:19:15.589224 10327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a69a4d7964d4220a080f04e1597a49c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a69a4d7964d4220a080f04e1597a49c" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43297 } }
I20260619 08:19:15.589366 10327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a69a4d7964d4220a080f04e1597a49c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:15.589665 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 10259
I20260619 08:19:15.589771 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:19:15.589977 4702 external_mini_cluster.cc:1428] Setting key c8003fbcf52ac2d30aeb2888704aa970
I20260619 08:19:15.590096 10327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a69a4d7964d4220a080f04e1597a49c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a69a4d7964d4220a080f04e1597a49c, State: Initialized, Role: FOLLOWER
I20260619 08:19:15.590325 10327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a69a4d7964d4220a080f04e1597a49c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a69a4d7964d4220a080f04e1597a49c" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43297 } }
I20260619 08:19:15.590435 10327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a69a4d7964d4220a080f04e1597a49c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:19:15.590476 10327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a69a4d7964d4220a080f04e1597a49c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:19:15.590524 10327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a69a4d7964d4220a080f04e1597a49c [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:15.591377 10327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a69a4d7964d4220a080f04e1597a49c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a69a4d7964d4220a080f04e1597a49c" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43297 } }
I20260619 08:19:15.591495 10327 leader_election.cc:302] T 00000000000000000000000000000000 P 1a69a4d7964d4220a080f04e1597a49c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a69a4d7964d4220a080f04e1597a49c; no voters:
I20260619 08:19:15.591660 10327 leader_election.cc:288] T 00000000000000000000000000000000 P 1a69a4d7964d4220a080f04e1597a49c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:19:15.591697 10332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a69a4d7964d4220a080f04e1597a49c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:15.591835 10332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a69a4d7964d4220a080f04e1597a49c [term 1 LEADER]: Becoming Leader. State: Replica: 1a69a4d7964d4220a080f04e1597a49c, State: Running, Role: LEADER
I20260619 08:19:15.591938 10327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a69a4d7964d4220a080f04e1597a49c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:19:15.591933 10332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a69a4d7964d4220a080f04e1597a49c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a69a4d7964d4220a080f04e1597a49c" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43297 } }
I20260619 08:19:15.593298 10333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a69a4d7964d4220a080f04e1597a49c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a69a4d7964d4220a080f04e1597a49c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a69a4d7964d4220a080f04e1597a49c" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43297 } } }
I20260619 08:19:15.593524 10334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a69a4d7964d4220a080f04e1597a49c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a69a4d7964d4220a080f04e1597a49c. Latest consensus state: current_term: 1 leader_uuid: "1a69a4d7964d4220a080f04e1597a49c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a69a4d7964d4220a080f04e1597a49c" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43297 } } }
I20260619 08:19:15.593652 10333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a69a4d7964d4220a080f04e1597a49c [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:15.593689 10334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a69a4d7964d4220a080f04e1597a49c [sys.catalog]: This master's current role is: LEADER
W20260619 08:19:15.593999 10347 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a69a4d7964d4220a080f04e1597a49c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:19:15.594054 10347 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:19:15.594128 10349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:19:15.594348 10349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:19:15.595985 10349 catalog_manager.cc:1347] Generated new cluster ID: bc5c4793fa214f95be3fa6bc71c52bda
I20260619 08:19:15.596038 10349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:19:15 dist-test-slave-d3n8 krb5kdc[10246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:15.603101 10331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:15.590521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60680 (local address 127.4.151.190:43297)
0619 08:19:15.590741 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:15.590750 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:15.590818 (+ 68us) server_negotiation.cc:407] Connection header received
0619 08:19:15.591266 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:15.591276 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:15.591344 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:15.591507 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:15.592436 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:15.594092 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:15.595135 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:15.595748 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:15.595863 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:15.598664 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:15.598668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:15.598668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:15.598681 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:15.601341 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:15.601793 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:15.601796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:15.601797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:15.601933 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:15.602135 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:15.602135 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:15.602136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:15.602472 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:15.602641 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:15.602714 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 08:19:15.602863 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":114,"threads_started":1}
I20260619 08:19:15.606480 10349 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:19:15.607185 10349 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:19:15.613605 10349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a69a4d7964d4220a080f04e1597a49c: Generated new TSK 0
I20260619 08:19:15.613838 10349 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.151.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
I20260619 08:19:15.640753 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43297
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:15.723938 10353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:15.724220 10353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:15.724264 10353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:15.725565 10353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:15.725631 10353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:15.725677 10353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:19:15.727221 10353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:15.727512 10353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:15.727757 10353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:15.732935 10361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:15.733052 10359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:15.732941 10358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:15.733223 10353 server_base.cc:1034] running on GCE node
I20260619 08:19:15.733526 10353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:15.733771 10353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:15.734977 10353 hybrid_clock.cc:648] HybridClock initialized: now 1781857155734957 us; error 47 us; skew 500 ppm
Jun 19 08:19:15 dist-test-slave-d3n8 krb5kdc[10246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857155, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:15.737793 10353 init.cc:377] Logged in from keytab as kudu/127.4.151.129@KRBTEST.COM (short username kudu)
I20260619 08:19:15.738344 10353 webserver.cc:460] Webserver started at http://127.4.151.129:39855/ using document root <none> and password file <none>
I20260619 08:19:15.738605 10353 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:15.738651 10353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:15.738782 10353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:15.739874 10353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "27dd22d85f954cffb5a4fbe82d5e6528"
format_stamp: "Formatted at 2026-06-19 08:19:15 on dist-test-slave-d3n8"
server_key: "111ddd620af3384d89107aea4e9ffa58"
server_key_iv: "92910c91dfbd4a6caa052909c96e4b6d"
server_key_version: "encryptionkey@0"
I20260619 08:19:15.740252 10353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "27dd22d85f954cffb5a4fbe82d5e6528"
format_stamp: "Formatted at 2026-06-19 08:19:15 on dist-test-slave-d3n8"
server_key: "111ddd620af3384d89107aea4e9ffa58"
server_key_iv: "92910c91dfbd4a6caa052909c96e4b6d"
server_key_version: "encryptionkey@0"
I20260619 08:19:15.741876 10353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 08:19:15.743202 10368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:15.743582 10353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:15.743664 10353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "27dd22d85f954cffb5a4fbe82d5e6528"
format_stamp: "Formatted at 2026-06-19 08:19:15 on dist-test-slave-d3n8"
server_key: "111ddd620af3384d89107aea4e9ffa58"
server_key_iv: "92910c91dfbd4a6caa052909c96e4b6d"
server_key_version: "encryptionkey@0"
I20260619 08:19:15.743774 10353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:15.759699 10353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:15.762060 10353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:15.762226 10353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:15.762493 10353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:15.762809 10353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:15.762844 10353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:15.762867 10353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:15.762902 10353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:15.769142 10353 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:38541
I20260619 08:19:15.769254 10481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:38541 every 8 connection(s)
I20260619 08:19:15.769588 10353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
Jun 19 08:19:15 dist-test-slave-d3n8 krb5kdc[10246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857155, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:15.776248 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 10353
I20260619 08:19:15.776362 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:19:15.776582 4702 external_mini_cluster.cc:1428] Setting key 3b37f74820d91267a33a50c064b5d072
I20260619 08:19:15.780488 10331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:15.770391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:41551 (local address 127.4.151.190:43297)
0619 08:19:15.770463 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:15.770509 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:15.770676 (+ 167us) server_negotiation.cc:407] Connection header received
0619 08:19:15.771656 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:15.771657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:15.771669 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:15.771722 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:15.772365 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:15.772896 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:15.774031 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:15.774246 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:15.774322 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:15.776976 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:15.776978 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:15.776978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:15.776987 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:15.778863 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:15.779395 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:15.779396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:15.779396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:15.779450 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:15.779768 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:15.779770 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:15.779770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:15.779896 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:15.779990 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:15.780334 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 08:19:15.780422 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 08:19:15.780735 10484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:15.770361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:43297 (local address 127.4.151.129:41551)
0619 08:19:15.770543 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:15.770593 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:15.771530 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:15.771791 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:15.771795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:15.772067 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:15.772263 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:15.772267 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:15.773000 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:15.773001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:15.773904 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:15.773907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:15.774343 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:15.774344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:15.774475 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:15.775105 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:15.775113 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:15.776832 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:15.778953 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:15.778960 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:15.778962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:15.779308 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:15.779520 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:15.779521 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:15.779521 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:15.779677 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:15.779970 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:15.779976 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:15.780156 (+ 180us) client_negotiation.cc:769] Sending connection context
0619 08:19:15.780281 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 08:19:15.780437 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":94,"threads_started":1}
I20260619 08:19:15.781250 10482 heartbeater.cc:347] Connected to a master server at 127.4.151.190:43297
I20260619 08:19:15.781347 10482 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:15.781580 10482 heartbeater.cc:511] Master 127.4.151.190:43297 requested a full tablet report, sending...
I20260619 08:19:15.782208 10291 ts_manager.cc:194] Registered new tserver with Master: 27dd22d85f954cffb5a4fbe82d5e6528 (127.4.151.129:38541)
I20260619 08:19:15.783207 10291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.129@KRBTEST.COM'} at 127.4.151.129:41551
WARNING: no policy specified for kudu/127.4.151.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
I20260619 08:19:15.804925 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43297
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:15.886848 10487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:15.887073 10487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:15.887096 10487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:15.888337 10487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:15.888388 10487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:15.888417 10487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:19:15.889923 10487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:15.890158 10487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:15.890393 10487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:15.896051 10492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:15.896171 10487 server_base.cc:1034] running on GCE node
W20260619 08:19:15.896045 10493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:15.896045 10495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:15.896595 10487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:15.896864 10487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:15.898152 10487 hybrid_clock.cc:648] HybridClock initialized: now 1781857155898120 us; error 41 us; skew 500 ppm
Jun 19 08:19:15 dist-test-slave-d3n8 krb5kdc[10246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857155, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:15.900918 10487 init.cc:377] Logged in from keytab as kudu/127.4.151.130@KRBTEST.COM (short username kudu)
I20260619 08:19:15.901471 10487 webserver.cc:460] Webserver started at http://127.4.151.130:43107/ using document root <none> and password file <none>
I20260619 08:19:15.901665 10487 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:15.901706 10487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:15.901796 10487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:15.902808 10487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "381f7a20334340dfacffe298abfd8975"
format_stamp: "Formatted at 2026-06-19 08:19:15 on dist-test-slave-d3n8"
server_key: "31dcd9cd4880722fc99bab53019741ff"
server_key_iv: "4585c03893eec237af3eb44292a472a9"
server_key_version: "encryptionkey@0"
I20260619 08:19:15.903131 10487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "381f7a20334340dfacffe298abfd8975"
format_stamp: "Formatted at 2026-06-19 08:19:15 on dist-test-slave-d3n8"
server_key: "31dcd9cd4880722fc99bab53019741ff"
server_key_iv: "4585c03893eec237af3eb44292a472a9"
server_key_version: "encryptionkey@0"
I20260619 08:19:15.904484 10487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:19:15.905368 10502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:15.905622 10487 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:15.905699 10487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "381f7a20334340dfacffe298abfd8975"
format_stamp: "Formatted at 2026-06-19 08:19:15 on dist-test-slave-d3n8"
server_key: "31dcd9cd4880722fc99bab53019741ff"
server_key_iv: "4585c03893eec237af3eb44292a472a9"
server_key_version: "encryptionkey@0"
I20260619 08:19:15.905758 10487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:15.913820 10487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:15.915755 10487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:15.915863 10487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:15.916049 10487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:15.916286 10487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:15.916316 10487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:15.916337 10487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:15.916348 10487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:15.922343 10487 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:46745
I20260619 08:19:15.922449 10615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:46745 every 8 connection(s)
I20260619 08:19:15.922664 10487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
Jun 19 08:19:15 dist-test-slave-d3n8 krb5kdc[10246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857155, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:15.930051 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 10487
I20260619 08:19:15.930159 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:19:15.930366 4702 external_mini_cluster.cc:1428] Setting key 1bf6f3e762aa5805e3b181792bbd6bd5
I20260619 08:19:15.932785 10331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:15.923386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:59761 (local address 127.4.151.190:43297)
0619 08:19:15.923451 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:15.923453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:15.923657 (+ 204us) server_negotiation.cc:407] Connection header received
0619 08:19:15.924617 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:15.924618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:15.924631 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:15.924683 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:15.925310 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:15.925873 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:15.926928 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:15.927139 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:15.927211 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:15.929310 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:15.929312 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:15.929312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:15.929319 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:15.930971 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:15.931491 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:15.931492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:15.931493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:15.931566 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:15.931918 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:15.931919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:15.931919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:15.932122 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:15.932191 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:15.932516 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 08:19:15.932632 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 08:19:15.932986 10618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:15.923389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:43297 (local address 127.4.151.130:59761)
0619 08:19:15.923538 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:15.923570 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:15.924502 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:15.924768 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:15.924772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:15.925025 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:15.925210 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:15.925214 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:15.926033 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:15.926034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:15.926810 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:15.926812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:15.927231 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:15.927232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:15.927349 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:15.927810 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:15.927818 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:15.929210 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:15.931046 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:15.931052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:15.931054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:15.931371 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:15.931645 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:15.931646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:15.931647 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:15.931801 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:15.932201 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:15.932206 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:15.932375 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 08:19:15.932501 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 08:19:15.932708 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260619 08:19:15.933552 10616 heartbeater.cc:347] Connected to a master server at 127.4.151.190:43297
I20260619 08:19:15.933662 10616 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:15.933837 10616 heartbeater.cc:511] Master 127.4.151.190:43297 requested a full tablet report, sending...
I20260619 08:19:15.934264 10291 ts_manager.cc:194] Registered new tserver with Master: 381f7a20334340dfacffe298abfd8975 (127.4.151.130:46745)
I20260619 08:19:15.934814 10291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.130@KRBTEST.COM'} at 127.4.151.130:59761
WARNING: no policy specified for kudu/127.4.151.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
I20260619 08:19:15.959966 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43297
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:16.042098 10621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:16.042358 10621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:16.042398 10621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:16.043597 10621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:16.043644 10621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:16.043689 10621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:19:16.045171 10621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:16.045440 10621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:16.045696 10621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:19:16.051141 10621 server_base.cc:1034] running on GCE node
W20260619 08:19:16.051111 10626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:16.051107 10629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:16.051090 10627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:16.051623 10621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:16.051867 10621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:16.053072 10621 hybrid_clock.cc:648] HybridClock initialized: now 1781857156053057 us; error 28 us; skew 500 ppm
Jun 19 08:19:16 dist-test-slave-d3n8 krb5kdc[10246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857156, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:16.055959 10621 init.cc:377] Logged in from keytab as kudu/127.4.151.131@KRBTEST.COM (short username kudu)
I20260619 08:19:16.056409 10621 webserver.cc:460] Webserver started at http://127.4.151.131:41707/ using document root <none> and password file <none>
I20260619 08:19:16.056653 10621 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:16.056720 10621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:16.056857 10621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:16.057972 10621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "22d10800f62f410abd53eb14c74ac9e6"
format_stamp: "Formatted at 2026-06-19 08:19:16 on dist-test-slave-d3n8"
server_key: "37e7b898d9d8f7424a2fa4d289e2a0eb"
server_key_iv: "deaf247228f6f1930c848274d404f24e"
server_key_version: "encryptionkey@0"
I20260619 08:19:16.058324 10621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "22d10800f62f410abd53eb14c74ac9e6"
format_stamp: "Formatted at 2026-06-19 08:19:16 on dist-test-slave-d3n8"
server_key: "37e7b898d9d8f7424a2fa4d289e2a0eb"
server_key_iv: "deaf247228f6f1930c848274d404f24e"
server_key_version: "encryptionkey@0"
I20260619 08:19:16.059697 10621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:19:16.060583 10636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:16.060820 10621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:16.060946 10621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "22d10800f62f410abd53eb14c74ac9e6"
format_stamp: "Formatted at 2026-06-19 08:19:16 on dist-test-slave-d3n8"
server_key: "37e7b898d9d8f7424a2fa4d289e2a0eb"
server_key_iv: "deaf247228f6f1930c848274d404f24e"
server_key_version: "encryptionkey@0"
I20260619 08:19:16.061022 10621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:16.075124 10621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:16.077299 10621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:16.077548 10621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:16.077781 10621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:16.078166 10621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:16.078218 10621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:16.078260 10621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:16.078286 10621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:16.084015 10621 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:41601
I20260619 08:19:16.084070 10749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:41601 every 8 connection(s)
I20260619 08:19:16.084362 10621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
I20260619 08:19:16.084890 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 10621
I20260619 08:19:16.085083 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:19:16.085300 4702 external_mini_cluster.cc:1428] Setting key 1dcd92b2f3f2dd6860058ef8a3c88ac1
Jun 19 08:19:16 dist-test-slave-d3n8 krb5kdc[10246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857156, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:16.094816 10331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:16.085091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:59321 (local address 127.4.151.190:43297)
0619 08:19:16.085142 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:16.085144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:16.085413 (+ 269us) server_negotiation.cc:407] Connection header received
0619 08:19:16.086713 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:16.086714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:16.086726 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:16.086769 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:16.087384 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:16.087884 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:16.088877 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:16.089111 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:16.089200 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:16.091603 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:16.091605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:16.091605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:16.091613 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:16.093289 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:16.093784 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:16.093785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:16.093785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:16.093846 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:16.094138 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:16.094139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:16.094139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:16.094260 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:16.094336 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:16.094627 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 08:19:16.094718 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 08:19:16.094985 10752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:16.085089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:43297 (local address 127.4.151.131:59321)
0619 08:19:16.085255 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:16.085290 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:16.086609 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:16.086834 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:16.086838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:16.087098 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:16.087281 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:16.087285 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:16.087986 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:16.087988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:16.088786 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:16.088788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:16.089212 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:16.089213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:16.089352 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:16.089982 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:16.089990 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:16.091506 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:16.093398 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:16.093404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:16.093406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:16.093696 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:16.093927 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:16.093927 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:16.093928 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:16.094060 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:16.094327 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:16.094332 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:16.094477 (+ 145us) client_negotiation.cc:769] Sending connection context
0619 08:19:16.094580 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 08:19:16.094734 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260619 08:19:16.095484 10750 heartbeater.cc:347] Connected to a master server at 127.4.151.190:43297
I20260619 08:19:16.095580 10750 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:16.095734 10750 heartbeater.cc:511] Master 127.4.151.190:43297 requested a full tablet report, sending...
I20260619 08:19:16.096127 10291 ts_manager.cc:194] Registered new tserver with Master: 22d10800f62f410abd53eb14c74ac9e6 (127.4.151.131:41601)
I20260619 08:19:16.096534 10291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.131@KRBTEST.COM'} at 127.4.151.131:59321
I20260619 08:19:16.097610 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/test-admin.keytab.
Jun 19 08:19:16 dist-test-slave-d3n8 krb5kdc[10246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:16.114719 4702 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 08:19:16.116389 10757 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:16.115614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38444 (local address 127.4.151.129:38541)
0619 08:19:16.115806 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:16.115810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:16.115821 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:19:16.115846 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:16.115847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:16.115861 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:16.115997 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:16.116279 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38444: BlockingRecv error: recv got EOF from 127.0.0.1:38444 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":100,"threads_started":1}
W20260619 08:19:16.116484 4702 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/krb5cc' of type FILE
Jun 19 08:19:16 dist-test-slave-d3n8 krb5kdc[10246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:16.117682 4702 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 08:19:16 dist-test-slave-d3n8 krb5kdc[10246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:16.118707 4702 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 08:19:16 dist-test-slave-d3n8 krb5kdc[10246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.129@KRBTEST.COM
I20260619 08:19:16.126842 10757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:16.119394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38456 (local address 127.4.151.129:38541)
0619 08:19:16.119458 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:16.119459 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:16.119466 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:19:16.119484 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:16.119485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:16.119494 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:16.119559 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:16.120031 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:16.120778 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:16.121577 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:16.121899 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:16.121990 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:16.124199 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:16.124206 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:16.124206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:16.124212 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:16.125740 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:16.126087 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:16.126089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:16.126089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:16.126141 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:16.126341 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:16.126342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:16.126342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:16.126490 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:16.126602 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:16.126701 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 08:19:16.126764 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 08:19:16.128999 10757 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:16.128406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38460 (local address 127.4.151.129:38541)
0619 08:19:16.128467 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:16.128469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:16.128523 (+ 54us) server_negotiation.cc:407] Connection header received
0619 08:19:16.128587 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:16.128588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:16.128598 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:16.128667 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:16.128947 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38460: BlockingRecv error: recv got EOF from 127.0.0.1:38460 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 08:19:16.129271 4702 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/krb5cc' of type FILE
Jun 19 08:19:16 dist-test-slave-d3n8 krb5kdc[10246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:16.130322 4702 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 08:19:16 dist-test-slave-d3n8 krb5kdc[10246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:16.131300 4702 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 08:19:16 dist-test-slave-d3n8 krb5kdc[10246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.129@KRBTEST.COM
I20260619 08:19:16.139051 10757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:16.132013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38464 (local address 127.4.151.129:38541)
0619 08:19:16.132063 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:16.132065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:16.132073 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:19:16.132114 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:16.132115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:16.132122 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:16.132161 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:16.132566 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:16.133096 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:16.133888 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:16.134160 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:16.134230 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:16.136413 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:16.136414 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:16.136414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:16.136420 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:16.137850 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:16.138249 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:16.138250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:16.138250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:16.138302 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:16.138559 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:16.138560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:16.138561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:16.138670 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:16.138768 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:16.138895 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 08:19:16.138972 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260619 08:19:16.140792 10757 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:16.140215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38468 (local address 127.4.151.129:38541)
0619 08:19:16.140283 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:16.140285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:16.140334 (+ 49us) server_negotiation.cc:407] Connection header received
0619 08:19:16.140399 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:16.140401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:16.140412 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:16.140472 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:16.140741 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38468: BlockingRecv error: recv got EOF from 127.0.0.1:38468 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 08:19:16.141048 4702 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781857130600961-4702-0/krb5kdc/krb5cc)
Jun 19 08:19:16 dist-test-slave-d3n8 krb5kdc[10246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:16.142226 4702 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 08:19:16 dist-test-slave-d3n8 krb5kdc[10246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:16.143195 4702 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 08:19:16 dist-test-slave-d3n8 krb5kdc[10246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.129@KRBTEST.COM
I20260619 08:19:16.151090 10757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:16.143981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38470 (local address 127.4.151.129:38541)
0619 08:19:16.144037 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:16.144039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:16.144048 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:19:16.144068 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:16.144068 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:16.144077 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:16.144138 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:16.144540 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:16.145019 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:16.145863 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:16.146082 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:16.146183 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:16.148543 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:16.148544 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:16.148544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:16.148551 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:16.149977 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:16.150345 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:16.150346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:16.150346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:16.150404 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:16.150589 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:16.150590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:16.150590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:16.150686 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:16.150756 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:16.150963 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 08:19:16.151028 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 08:19:16.151819 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 10353
I20260619 08:19:16.157055 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 10487
I20260619 08:19:16.161257 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 10621
I20260619 08:19:16.165762 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 10259
2026-06-19T08:19:16Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1888 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:19:16 dist-test-slave-d3n8 krb5kdc[10770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:19:16 dist-test-slave-d3n8 krb5kdc[10770](info): set up 2 sockets
Jun 19 08:19:16 dist-test-slave-d3n8 krb5kdc[10770](info): commencing operation
krb5kdc: starting...
W20260619 08:19:17.205785 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:19:17 dist-test-slave-d3n8 krb5kdc[10770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:19:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:19:17Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.190.keytab.
Entry for principal oryx/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.190.keytab.
I20260619 08:19:17.333242 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:44597
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.190.keytab
--principal=oryx/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:40861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:44597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:17.413627 10783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:17.413839 10783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:17.413877 10783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:17.415078 10783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:19:17.415124 10783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:17.415139 10783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:17.415153 10783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:19:17.415164 10783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:19:17.416714 10783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:40861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:44597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.190.keytab
--principal=oryx/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:44597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:17.416982 10783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:17.417236 10783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:17.422237 10788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:17.422267 10791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:17.422248 10789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:17.422538 10783 server_base.cc:1034] running on GCE node
I20260619 08:19:17.422822 10783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:17.423106 10783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:17.424264 10783 hybrid_clock.cc:648] HybridClock initialized: now 1781857157424254 us; error 31 us; skew 500 ppm
Jun 19 08:19:17 dist-test-slave-d3n8 krb5kdc[10770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857157, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:17.430811 10783 init.cc:377] Logged in from keytab as oryx/127.4.151.190@KRBTEST.COM (short username oryx)
I20260619 08:19:17.431314 10783 webserver.cc:460] Webserver started at http://127.4.151.190:36043/ using document root <none> and password file <none>
I20260619 08:19:17.431531 10783 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:17.431573 10783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:17.431665 10783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:17.432685 10783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "d6dd1d19eb084efbbf3cca9f38d54286"
format_stamp: "Formatted at 2026-06-19 08:19:17 on dist-test-slave-d3n8"
server_key: "2161228aab90ca590732ab3cc3a4c2da"
server_key_iv: "c9db5d5aa4c9c179a2866ecfb5e5aeab"
server_key_version: "encryptionkey@0"
I20260619 08:19:17.432983 10783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "d6dd1d19eb084efbbf3cca9f38d54286"
format_stamp: "Formatted at 2026-06-19 08:19:17 on dist-test-slave-d3n8"
server_key: "2161228aab90ca590732ab3cc3a4c2da"
server_key_iv: "c9db5d5aa4c9c179a2866ecfb5e5aeab"
server_key_version: "encryptionkey@0"
I20260619 08:19:17.434466 10783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:19:17.435302 10798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:17.435534 10783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:17.435640 10783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "d6dd1d19eb084efbbf3cca9f38d54286"
format_stamp: "Formatted at 2026-06-19 08:19:17 on dist-test-slave-d3n8"
server_key: "2161228aab90ca590732ab3cc3a4c2da"
server_key_iv: "c9db5d5aa4c9c179a2866ecfb5e5aeab"
server_key_version: "encryptionkey@0"
I20260619 08:19:17.435724 10783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:17.448238 10783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:17.450642 10783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:17.450790 10783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:17.454721 10783 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:44597
I20260619 08:19:17.454752 10850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:44597 every 8 connection(s)
I20260619 08:19:17.455108 10783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:19:17.456123 10851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:17.458272 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 10783
I20260619 08:19:17.458406 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:19:17.458621 4702 external_mini_cluster.cc:1428] Setting key 0b4b08a081bae0732d188116e98ee8f0
I20260619 08:19:17.458637 10851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286: Bootstrap starting.
I20260619 08:19:17.459337 10851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:17.459656 10851 log.cc:824] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:17.460423 10851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286: No bootstrap required, opened a new log
I20260619 08:19:17.462697 10851 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } }
I20260619 08:19:17.462868 10851 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:17.462899 10851 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6dd1d19eb084efbbf3cca9f38d54286, State: Initialized, Role: FOLLOWER
I20260619 08:19:17.463030 10851 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } }
I20260619 08:19:17.463155 10851 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:19:17.463181 10851 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:19:17.463258 10851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:17.463860 10851 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } }
I20260619 08:19:17.463976 10851 leader_election.cc:302] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6dd1d19eb084efbbf3cca9f38d54286; no voters:
I20260619 08:19:17.464183 10851 leader_election.cc:288] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:19:17.464205 10856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:17.464316 10856 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [term 1 LEADER]: Becoming Leader. State: Replica: d6dd1d19eb084efbbf3cca9f38d54286, State: Running, Role: LEADER
I20260619 08:19:17.464395 10856 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } }
I20260619 08:19:17.464450 10851 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:19:17.464769 10856 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } }
I20260619 08:19:17.464833 10856 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:17.464967 10856 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6dd1d19eb084efbbf3cca9f38d54286. Latest consensus state: current_term: 1 leader_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } }
I20260619 08:19:17.465022 10856 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:17.465833 10861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 19 08:19:17 dist-test-slave-d3n8 krb5kdc[10770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM
I20260619 08:19:17.466341 10861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:19:17.469815 10861 catalog_manager.cc:1347] Generated new cluster ID: edb1d5b47cc64477b0439e888897e14a
I20260619 08:19:17.469878 10861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:19:17.470593 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:17.459201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33356 (local address 127.4.151.190:44597)
0619 08:19:17.459441 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:17.459452 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:17.459519 (+ 67us) server_negotiation.cc:407] Connection header received
0619 08:19:17.460030 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:17.460038 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:17.460086 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:17.460206 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:17.460999 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:17.461996 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:17.463095 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:17.463631 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:17.463726 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:17.466866 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:17.466871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:17.466871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:17.466884 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:17.468791 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:17.469199 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:17.469202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:17.469202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:17.469300 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:17.469534 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:17.469535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:17.469536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:17.469833 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:17.470006 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:17.470246 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 08:19:17.470386 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":90,"threads_started":1}
I20260619 08:19:17.478806 10861 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:19:17.479507 10861 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:19:17.490980 10861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286: Generated new TSK 0
I20260619 08:19:17.491178 10861 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.151.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.151.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.151.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.129.keytab.
Entry for principal oryx/127.4.151.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.129.keytab.
I20260619 08:19:17.521459 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:44597
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.129.keytab
--principal=oryx/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:40861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:17.612035 10877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:17.612242 10877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:17.612267 10877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:17.613547 10877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:17.613591 10877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:17.613619 10877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:19:17.615139 10877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:40861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.129.keytab
--principal=oryx/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:44597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:17.615389 10877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:17.615646 10877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:17.621680 10882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:17.621661 10883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:17.621658 10885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:17.621907 10877 server_base.cc:1034] running on GCE node
I20260619 08:19:17.622217 10877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:17.622463 10877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:17.623633 10877 hybrid_clock.cc:648] HybridClock initialized: now 1781857157623628 us; error 39 us; skew 500 ppm
Jun 19 08:19:17 dist-test-slave-d3n8 krb5kdc[10770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857157, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:17.626398 10877 init.cc:377] Logged in from keytab as oryx/127.4.151.129@KRBTEST.COM (short username oryx)
I20260619 08:19:17.626946 10877 webserver.cc:460] Webserver started at http://127.4.151.129:38461/ using document root <none> and password file <none>
I20260619 08:19:17.627279 10877 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:17.627351 10877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:17.627506 10877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:17.628579 10877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "130acc74148042b68fad3073fe31193a"
format_stamp: "Formatted at 2026-06-19 08:19:17 on dist-test-slave-d3n8"
server_key: "01fb153b3119cd54434167d1e123c2cf"
server_key_iv: "b5018a61aff0b78a3278b1f14ac48864"
server_key_version: "encryptionkey@0"
I20260619 08:19:17.628940 10877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "130acc74148042b68fad3073fe31193a"
format_stamp: "Formatted at 2026-06-19 08:19:17 on dist-test-slave-d3n8"
server_key: "01fb153b3119cd54434167d1e123c2cf"
server_key_iv: "b5018a61aff0b78a3278b1f14ac48864"
server_key_version: "encryptionkey@0"
I20260619 08:19:17.630390 10877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:19:17.631287 10892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:17.631536 10877 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:17.631659 10877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "130acc74148042b68fad3073fe31193a"
format_stamp: "Formatted at 2026-06-19 08:19:17 on dist-test-slave-d3n8"
server_key: "01fb153b3119cd54434167d1e123c2cf"
server_key_iv: "b5018a61aff0b78a3278b1f14ac48864"
server_key_version: "encryptionkey@0"
I20260619 08:19:17.631734 10877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:17.650098 10877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:17.652372 10877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:17.652585 10877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:17.652833 10877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:17.653168 10877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:17.653223 10877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:17.653258 10877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:17.653286 10877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:17.659399 10877 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:41629
I20260619 08:19:17.659446 11005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:41629 every 8 connection(s)
I20260619 08:19:17.659755 10877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
Jun 19 08:19:17 dist-test-slave-d3n8 krb5kdc[10770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857157, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM
I20260619 08:19:17.667387 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 10877
I20260619 08:19:17.667500 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:19:17.667695 4702 external_mini_cluster.cc:1428] Setting key 2bd13f111b33e77e696b4dfbcb09e8e5
I20260619 08:19:17.670243 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:17.660532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:34271 (local address 127.4.151.190:44597)
0619 08:19:17.660580 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:17.660587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:17.660679 (+ 92us) server_negotiation.cc:407] Connection header received
0619 08:19:17.661186 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:17.661187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:17.661200 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:17.661265 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:17.662388 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:17.662912 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:17.663974 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:17.664270 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:17.664322 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:17.666792 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:17.666794 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:17.666794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:17.666801 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:17.668737 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:17.669200 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:17.669201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:17.669202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:17.669267 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:17.669569 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:17.669570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:17.669571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:17.669696 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:17.669775 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:17.670058 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 08:19:17.670152 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:19:17.670449 11008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:17.660404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.129:34271)
0619 08:19:17.660567 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:17.660609 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:17.661081 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:17.661347 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:17.661350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:17.662079 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:17.662290 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:17.662295 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:17.663083 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:17.663084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:17.663882 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:17.663884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:17.664331 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:17.664331 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:17.664447 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:17.665051 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:17.665060 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:17.666678 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:17.668833 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:17.668841 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:17.668843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:17.669105 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:17.669338 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:17.669339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:17.669339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:17.669497 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:17.669765 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:17.669772 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:17.669920 (+ 148us) client_negotiation.cc:769] Sending connection context
0619 08:19:17.670026 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 08:19:17.670191 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260619 08:19:17.670909 11006 heartbeater.cc:347] Connected to a master server at 127.4.151.190:44597
I20260619 08:19:17.671022 11006 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:17.671231 11006 heartbeater.cc:511] Master 127.4.151.190:44597 requested a full tablet report, sending...
I20260619 08:19:17.671737 10815 ts_manager.cc:194] Registered new tserver with Master: 130acc74148042b68fad3073fe31193a (127.4.151.129:41629)
I20260619 08:19:17.672376 10815 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.151.129@KRBTEST.COM'} at 127.4.151.129:34271
WARNING: no policy specified for oryx/127.4.151.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.151.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.151.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.130.keytab.
Entry for principal oryx/127.4.151.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.130.keytab.
I20260619 08:19:17.697114 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:44597
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.130.keytab
--principal=oryx/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:40861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:17.780254 11011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:17.780459 11011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:17.780483 11011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:17.781764 11011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:17.781806 11011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:17.781834 11011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:19:17.783310 11011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:40861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.130.keytab
--principal=oryx/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:44597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:17.783562 11011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:17.783810 11011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:17.789249 11019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:17.789222 11011 server_base.cc:1034] running on GCE node
W20260619 08:19:17.789165 11017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:17.789165 11016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:17.789853 11011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:17.790096 11011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:17.791253 11011 hybrid_clock.cc:648] HybridClock initialized: now 1781857157791244 us; error 27 us; skew 500 ppm
Jun 19 08:19:17 dist-test-slave-d3n8 krb5kdc[10770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857157, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:17.793915 11011 init.cc:377] Logged in from keytab as oryx/127.4.151.130@KRBTEST.COM (short username oryx)
I20260619 08:19:17.794405 11011 webserver.cc:460] Webserver started at http://127.4.151.130:43689/ using document root <none> and password file <none>
I20260619 08:19:17.794618 11011 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:17.794662 11011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:17.794754 11011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:17.795794 11011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "9817b0faddf74855b5625fa520c160d7"
format_stamp: "Formatted at 2026-06-19 08:19:17 on dist-test-slave-d3n8"
server_key: "eeaa5556caf381b3699a908939e5aad3"
server_key_iv: "5c23de858e3e0402fea84b103544de51"
server_key_version: "encryptionkey@0"
I20260619 08:19:17.796094 11011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "9817b0faddf74855b5625fa520c160d7"
format_stamp: "Formatted at 2026-06-19 08:19:17 on dist-test-slave-d3n8"
server_key: "eeaa5556caf381b3699a908939e5aad3"
server_key_iv: "5c23de858e3e0402fea84b103544de51"
server_key_version: "encryptionkey@0"
I20260619 08:19:17.797519 11011 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:19:17.798319 11026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:17.798652 11011 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:17.798748 11011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "9817b0faddf74855b5625fa520c160d7"
format_stamp: "Formatted at 2026-06-19 08:19:17 on dist-test-slave-d3n8"
server_key: "eeaa5556caf381b3699a908939e5aad3"
server_key_iv: "5c23de858e3e0402fea84b103544de51"
server_key_version: "encryptionkey@0"
I20260619 08:19:17.798810 11011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:17.806562 11011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:17.808888 11011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:17.809002 11011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:17.809186 11011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:17.809506 11011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:17.809542 11011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:17.809566 11011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:17.809578 11011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:17.815546 11011 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:45771
I20260619 08:19:17.815622 11139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:45771 every 8 connection(s)
I20260619 08:19:17.815950 11011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
Jun 19 08:19:17 dist-test-slave-d3n8 krb5kdc[10770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857157, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM
I20260619 08:19:17.822417 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 11011
I20260619 08:19:17.822520 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:19:17.822687 4702 external_mini_cluster.cc:1428] Setting key c4807f7ce0d9ab9943b0baa313cf80f9
I20260619 08:19:17.826210 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:17.816740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:50019 (local address 127.4.151.190:44597)
0619 08:19:17.816793 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:17.816796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:17.817121 (+ 325us) server_negotiation.cc:407] Connection header received
0619 08:19:17.817723 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:17.817724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:17.817735 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:17.817797 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:17.818814 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:17.819389 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:17.820378 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:17.820580 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:17.820638 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:17.823072 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:17.823073 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:17.823074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:17.823080 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:17.824566 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:17.825112 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:17.825113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:17.825113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:17.825184 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:17.825470 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:17.825471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:17.825472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:17.825591 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:17.825673 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:17.826069 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 08:19:17.826145 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:19:17.826400 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:17.816729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.130:50019)
0619 08:19:17.816891 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:17.816974 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:17.817607 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:17.817869 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:17.817872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:17.818452 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:17.818692 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:17.818697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:17.819479 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:17.819480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:17.820291 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:17.820293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:17.820656 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:17.820656 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:17.820755 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:17.821225 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:17.821234 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:17.822924 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:17.824695 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:17.824702 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:17.824704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:17.824936 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:17.825254 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:17.825255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:17.825255 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:17.825407 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:17.825663 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:17.825669 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:17.825832 (+ 163us) client_negotiation.cc:769] Sending connection context
0619 08:19:17.825935 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 08:19:17.826126 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260619 08:19:17.826967 11140 heartbeater.cc:347] Connected to a master server at 127.4.151.190:44597
I20260619 08:19:17.827097 11140 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:17.827296 11140 heartbeater.cc:511] Master 127.4.151.190:44597 requested a full tablet report, sending...
I20260619 08:19:17.827705 10815 ts_manager.cc:194] Registered new tserver with Master: 9817b0faddf74855b5625fa520c160d7 (127.4.151.130:45771)
I20260619 08:19:17.828096 10815 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.151.130@KRBTEST.COM'} at 127.4.151.130:50019
WARNING: no policy specified for oryx/127.4.151.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.151.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.151.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.131.keytab.
Entry for principal oryx/127.4.151.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.131.keytab.
I20260619 08:19:17.853626 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:44597
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.131.keytab
--principal=oryx/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:40861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:17.940088 11145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:17.940318 11145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:17.940343 11145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:17.941540 11145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:17.941583 11145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:17.941609 11145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:19:17.943113 11145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:40861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.131.keytab
--principal=oryx/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:44597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:17.943364 11145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:17.943594 11145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:17.949029 11151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:17.949096 11150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:17.949029 11153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:17.949293 11145 server_base.cc:1034] running on GCE node
I20260619 08:19:17.949561 11145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:17.949842 11145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:17.951072 11145 hybrid_clock.cc:648] HybridClock initialized: now 1781857157951020 us; error 27 us; skew 500 ppm
Jun 19 08:19:17 dist-test-slave-d3n8 krb5kdc[10770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857157, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:17.954064 11145 init.cc:377] Logged in from keytab as oryx/127.4.151.131@KRBTEST.COM (short username oryx)
I20260619 08:19:17.954553 11145 webserver.cc:460] Webserver started at http://127.4.151.131:45983/ using document root <none> and password file <none>
I20260619 08:19:17.954802 11145 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:17.954849 11145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:17.955025 11145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:17.956110 11145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "f5de23594c5a401cbb3b42642de5fe2e"
format_stamp: "Formatted at 2026-06-19 08:19:17 on dist-test-slave-d3n8"
server_key: "0c10a82d585dc5f6aaad0237d8472994"
server_key_iv: "ffaa232816d2b593e8331eef4cb2002b"
server_key_version: "encryptionkey@0"
I20260619 08:19:17.956486 11145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "f5de23594c5a401cbb3b42642de5fe2e"
format_stamp: "Formatted at 2026-06-19 08:19:17 on dist-test-slave-d3n8"
server_key: "0c10a82d585dc5f6aaad0237d8472994"
server_key_iv: "ffaa232816d2b593e8331eef4cb2002b"
server_key_version: "encryptionkey@0"
I20260619 08:19:17.958079 11145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:19:17.958962 11160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:17.959188 11145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:17.959300 11145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "f5de23594c5a401cbb3b42642de5fe2e"
format_stamp: "Formatted at 2026-06-19 08:19:17 on dist-test-slave-d3n8"
server_key: "0c10a82d585dc5f6aaad0237d8472994"
server_key_iv: "ffaa232816d2b593e8331eef4cb2002b"
server_key_version: "encryptionkey@0"
I20260619 08:19:17.959380 11145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:18.013675 11145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:18.015839 11145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:18.016016 11145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:18.016264 11145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:18.016577 11145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:18.016630 11145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:18.016671 11145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:18.016700 11145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:18.022882 11145 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:39947
I20260619 08:19:18.022969 11273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:39947 every 8 connection(s)
I20260619 08:19:18.023456 11145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
I20260619 08:19:18.029619 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 11145
I20260619 08:19:18.029737 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:19:18.029995 4702 external_mini_cluster.cc:1428] Setting key 263a82077277efdc8087281df26d03be
Jun 19 08:19:18 dist-test-slave-d3n8 krb5kdc[10770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857157, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM
I20260619 08:19:18.034163 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.024290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:43475 (local address 127.4.151.190:44597)
0619 08:19:18.024357 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.024360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.024559 (+ 199us) server_negotiation.cc:407] Connection header received
0619 08:19:18.025074 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.025075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.025093 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.025156 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:18.026328 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.026835 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.027973 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.028168 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.028231 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.030987 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:18.030988 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:18.030989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:18.030997 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:18.032548 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.033052 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.033054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.033054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.033117 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.033447 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.033448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.033449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.033594 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:18.033653 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:18.033913 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 08:19:18.034098 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 08:19:18.034315 11276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.024288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.131:43475)
0619 08:19:18.024447 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:18.024483 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:18.024948 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:18.025281 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:18.025285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:18.025970 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:18.026209 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.026215 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.026942 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.026943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.027806 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.027808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.028249 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.028250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.028354 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.029040 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:18.029049 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:18.030829 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:18.032649 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.032655 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.032657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.032885 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.033202 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.033203 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.033203 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.033317 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.033665 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:18.033669 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:18.033789 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 08:19:18.033889 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 08:19:18.034115 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260619 08:19:18.034811 11274 heartbeater.cc:347] Connected to a master server at 127.4.151.190:44597
I20260619 08:19:18.034893 11274 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:18.035090 11274 heartbeater.cc:511] Master 127.4.151.190:44597 requested a full tablet report, sending...
I20260619 08:19:18.035431 10815 ts_manager.cc:194] Registered new tserver with Master: f5de23594c5a401cbb3b42642de5fe2e (127.4.151.131:39947)
I20260619 08:19:18.035789 10815 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.151.131@KRBTEST.COM'} at 127.4.151.131:43475
I20260619 08:19:18.041875 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.151.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.151.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.151.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.189.keytab.
Entry for principal oryx/127.4.151.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.189.keytab.
I20260619 08:19:18.069645 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.189:44003
--webserver_interface=127.4.151.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.189.keytab
--principal=oryx/127.4.151.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:40861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:44597,127.4.151.189:44003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:18.150996 11279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:18.151212 11279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:18.151234 11279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:18.152384 11279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:19:18.152427 11279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:18.152441 11279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:18.152453 11279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:19:18.152467 11279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:19:18.154021 11279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:40861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:44597,127.4.151.189:44003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.189.keytab
--principal=oryx/127.4.151.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.189:44003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.151.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
W20260619 08:19:18.154160 11279 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.4.151.190:44597,127.4.151.189:44003'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 08:19:18.154309 11279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:18.154551 11279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:19:18.159401 11279 server_base.cc:1034] running on GCE node
W20260619 08:19:18.159377 11284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:18.159363 11287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:18.159379 11285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:18.159770 11279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:18.160007 11279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:18.161142 11279 hybrid_clock.cc:648] HybridClock initialized: now 1781857158161133 us; error 27 us; skew 500 ppm
Jun 19 08:19:18 dist-test-slave-d3n8 krb5kdc[10770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857158, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:18.163812 11279 init.cc:377] Logged in from keytab as oryx/127.4.151.189@KRBTEST.COM (short username oryx)
I20260619 08:19:18.164259 11279 webserver.cc:460] Webserver started at http://127.4.151.189:46233/ using document root <none> and password file <none>
I20260619 08:19:18.164623 11279 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:18.164702 11279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:18.164839 11279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:18.166113 11279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/data/instance:
uuid: "bed09c04d0564fecbde79ff3e9cfe92f"
format_stamp: "Formatted at 2026-06-19 08:19:18 on dist-test-slave-d3n8"
server_key: "1326f3adc89d49190c46023d36acd2b3"
server_key_iv: "9f05a61dd254d82256d915618fa055f1"
server_key_version: "encryptionkey@0"
I20260619 08:19:18.166491 11279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/wal/instance:
uuid: "bed09c04d0564fecbde79ff3e9cfe92f"
format_stamp: "Formatted at 2026-06-19 08:19:18 on dist-test-slave-d3n8"
server_key: "1326f3adc89d49190c46023d36acd2b3"
server_key_iv: "9f05a61dd254d82256d915618fa055f1"
server_key_version: "encryptionkey@0"
I20260619 08:19:18.167804 11279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:19:18.168534 11294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:18.168706 11279 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:19:18.168804 11279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/wal
uuid: "bed09c04d0564fecbde79ff3e9cfe92f"
format_stamp: "Formatted at 2026-06-19 08:19:18 on dist-test-slave-d3n8"
server_key: "1326f3adc89d49190c46023d36acd2b3"
server_key_iv: "9f05a61dd254d82256d915618fa055f1"
server_key_version: "encryptionkey@0"
I20260619 08:19:18.168888 11279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:18.175909 11279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:18.178243 11279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:18.178354 11279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:18.182277 11279 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.189:44003
I20260619 08:19:18.182309 11346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.189:44003 every 8 connection(s)
I20260619 08:19:18.182658 11279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/data/info.pb
I20260619 08:19:18.183645 11347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:18.184629 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 11279
I20260619 08:19:18.185616 11347 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 08:19:18 dist-test-slave-d3n8 krb5kdc[10770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857158, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.189@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM
Jun 19 08:19:18 dist-test-slave-d3n8 krb5kdc[10770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.151.189@KRBTEST.COM
I20260619 08:19:18.195477 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.186737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33370 (local address 127.4.151.190:44597)
0619 08:19:18.186786 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.186788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.187006 (+ 218us) server_negotiation.cc:407] Connection header received
0619 08:19:18.187037 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.187038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.187051 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.187121 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:18.187755 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.188259 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.189184 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.189411 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.189477 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.191824 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:18.191825 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:18.191826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:18.191834 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:18.193633 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.194083 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.194084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.194085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.194149 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.194498 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.194499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.194499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.194668 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:18.194744 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:18.195273 (+ 529us) server_negotiation.cc:299] Negotiation successful
0619 08:19:18.195360 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 08:19:18.195636 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.186668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.0.0.1:33370)
0619 08:19:18.186839 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:18.186871 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:18.186957 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:18.187169 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:18.187172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:18.187530 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:18.187639 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.187644 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.188363 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.188365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.189075 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.189077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.189492 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.189493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.189601 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.190110 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:18.190117 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:18.191728 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:18.193724 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.193729 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.193731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.193994 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.194248 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.194249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.194249 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.194380 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.194751 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:18.194757 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:18.195132 (+ 375us) client_negotiation.cc:769] Sending connection context
0619 08:19:18.195218 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 08:19:18.195379 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260619 08:19:18.195727 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.185367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51332 (local address 127.4.151.189:44003)
0619 08:19:18.185591 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.185601 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.185632 (+ 31us) server_negotiation.cc:407] Connection header received
0619 08:19:18.186198 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.186207 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.186267 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.186431 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:18.187228 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.188282 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.189132 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.189570 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.189640 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.192396 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:18.192404 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:18.192404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:18.192412 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:18.194322 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.194694 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.194697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.194697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.194755 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.195082 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.195083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.195083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.195235 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:18.195396 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:18.195520 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 08:19:18.195597 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":96,"threads_started":1}
I20260619 08:19:18.196372 11347 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 08:19:18 dist-test-slave-d3n8 krb5kdc[10770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857158, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.189@KRBTEST.COM for oryx/127.4.151.189@KRBTEST.COM
I20260619 08:19:18.203498 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.196699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:44003 (local address 127.0.0.1:51336)
0619 08:19:18.196738 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:18.196749 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:18.196787 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:18.197086 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:18.197087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:18.197204 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:18.197237 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.197238 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.198002 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.198002 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.198484 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.198485 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.198850 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.198851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.198977 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.199371 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:18.199375 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:18.200603 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:18.202438 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.202439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.202440 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.202621 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.202804 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.202805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.202805 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.202869 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.203209 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:18.203213 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:18.203249 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 08:19:18.203321 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 08:19:18.203429 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260619 08:19:18.203562 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.196767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51336 (local address 127.4.151.189:44003)
0619 08:19:18.196885 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.196887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.196894 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:19:18.196910 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.196911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.196918 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.196970 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:18.197328 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.197889 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.198557 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.198753 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.198814 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.200697 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:18.200698 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:18.200699 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:18.200705 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:18.202289 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.202692 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.202693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.202693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.202742 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.202963 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.202964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.202964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.203112 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:18.203177 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:18.203358 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 08:19:18.203479 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":92}
I20260619 08:19:18.205134 11347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f: Bootstrap starting.
I20260619 08:19:18.206094 11347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:18.206391 11347 log.cc:824] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:18.207298 11347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f: No bootstrap required, opened a new log
I20260619 08:19:18.208763 11347 raft_consensus.cc:348] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } }
I20260619 08:19:18.208894 11347 raft_consensus.cc:374] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:18.208918 11347 raft_consensus.cc:729] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed09c04d0564fecbde79ff3e9cfe92f, State: Initialized, Role: FOLLOWER
I20260619 08:19:18.209173 11347 consensus_queue.cc:260] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } }
I20260619 08:19:18.209527 11347 sys_catalog.cc:564] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:19:18.209522 11353 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } } }
I20260619 08:19:18.209681 11353 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [sys.catalog]: This master's current role is: FOLLOWER
W20260619 08:19:18.211396 11364 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:19:18.211462 11364 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:19:18.213459 11279 master_runner.cc:428] Detected that this master bed09c04d0564fecbde79ff3e9cfe92f is joining an existing cluster
I20260619 08:19:18.213505 11279 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.151.189:44003
I20260619 08:19:18.223100 11371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.214729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:44003 (local address 127.0.0.1:51338)
0619 08:19:18.214892 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:18.214904 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:18.214954 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:18.215155 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:18.215155 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:18.215473 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:18.215578 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.215581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.216498 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.216498 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.217558 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.217561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.218532 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.218533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.218622 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.219034 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:18.219038 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:18.219457 (+ 419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:18.221844 (+ 2387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.221845 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.221845 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.222080 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.222319 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.222320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.222321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.222376 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.222669 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:18.222670 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:18.222716 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 08:19:18.222811 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 08:19:18.222938 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":37,"thread_start_us":110,"threads_started":1}
I20260619 08:19:18.223129 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.214735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33372 (local address 127.4.151.190:44597)
0619 08:19:18.214785 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.214787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.214956 (+ 169us) server_negotiation.cc:407] Connection header received
0619 08:19:18.215229 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.215230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.215242 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.215307 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:18.215761 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.216334 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.217651 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.217978 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.218058 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.219492 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:18.219494 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:18.219494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:18.219502 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:18.221672 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.222214 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.222215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.222215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.222275 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.222524 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.222525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.222525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.222684 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:18.222753 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:18.222910 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 08:19:18.223005 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 08:19:18.223163 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.214932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51338 (local address 127.4.151.189:44003)
0619 08:19:18.214991 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.214993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.214999 (+ 6us) server_negotiation.cc:407] Connection header received
0619 08:19:18.215019 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.215020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.215030 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.215107 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:18.215672 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.216423 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.218138 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.218444 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.218516 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.219578 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:18.219579 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:18.219579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:18.219586 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:18.221781 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.222179 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.222180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.222180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.222240 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.222430 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.222431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.222431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.222591 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:18.222668 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:18.222845 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 08:19:18.223015 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":41}
I20260619 08:19:18.223395 11370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.214671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.0.0.1:33372)
0619 08:19:18.214871 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:18.214886 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:18.215145 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:18.215320 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:18.215321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:18.215562 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:18.215637 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.215638 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.216431 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.216432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.217558 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.217561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.218068 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.218069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.218141 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.218824 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:18.218830 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:18.219382 (+ 552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:18.221801 (+ 2419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.221802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.221802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.222107 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.222344 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.222345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.222345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.222410 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.222741 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:18.222742 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:18.222783 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 08:19:18.223213 (+ 430us) client_negotiation.cc:240] Negotiation successful
0619 08:19:18.223327 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":44,"thread_start_us":93,"threads_started":1}
Jun 19 08:19:18 dist-test-slave-d3n8 krb5kdc[10770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857157, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.190@KRBTEST.COM for oryx/127.4.151.189@KRBTEST.COM
I20260619 08:19:18.234123 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.226491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51354 (local address 127.4.151.189:44003)
0619 08:19:18.226532 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.226534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.226695 (+ 161us) server_negotiation.cc:407] Connection header received
0619 08:19:18.226794 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.226795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.226807 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.226868 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:18.227330 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.227893 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.228760 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.229002 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.229082 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.231223 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:18.231225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:18.231225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:18.231232 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:18.232824 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.233223 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.233224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.233224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.233281 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.233534 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.233535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.233535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.233661 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:18.233734 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:18.233949 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 08:19:18.234022 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 08:19:18.234328 11372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.226418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:44003 (local address 127.0.0.1:51354)
0619 08:19:18.226589 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:18.226629 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:18.226722 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:18.226874 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:18.226877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:18.227185 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:18.227240 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.227242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.227979 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.227981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.228648 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.228650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.229095 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.229096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.229187 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.229678 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:18.229683 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:18.231135 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:18.232918 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.232925 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.232927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.233142 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.233348 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.233349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.233349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.233474 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.233750 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:18.233757 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:18.233805 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 08:19:18.233902 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 08:19:18.234101 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":67,"threads_started":1}
I20260619 08:19:18.235069 10815 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.151.189:44003: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f"
member_type: NON_VOTER
last_known_addr {
host: "127.4.151.189"
port: 44003
}
attrs {
promote: true
}
}
dest_uuid: "d6dd1d19eb084efbbf3cca9f38d54286"
cas_config_opid_index: -1
I20260619 08:19:18.235388 10815 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: NON_VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: true } }
I20260619 08:19:18.235880 11374 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER bed09c04d0564fecbde79ff3e9cfe92f (127.4.151.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: NON_VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: true } } }
I20260619 08:19:18.236213 11321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:18.237026 11374 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.151.189:44003
I20260619 08:19:18.237107 11321 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 1 FOLLOWER]: Refusing update from remote peer d6dd1d19eb084efbbf3cca9f38d54286: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 08:19:18.237156 11374 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: NON_VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: true } } }
I20260619 08:19:18.237210 11374 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:18.237300 11279 master.cc:537] Master@127.4.151.189:44003 shutting down...
I20260619 08:19:18.237475 11373 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: NON_VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 08:19:18.238186 10801 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 08:19:18.238623 10801 consensus_peers.cc:487] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 -> Peer bed09c04d0564fecbde79ff3e9cfe92f (127.4.151.189:44003): Couldn't send request to peer bed09c04d0564fecbde79ff3e9cfe92f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 08:19:18.240144 11279 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:19:18.240229 11279 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:18.240283 11279 tablet_replica.cc:330] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f: stopping tablet replica
I20260619 08:19:18.252969 11279 master.cc:552] Master@127.4.151.189:44003 shutdown complete.
I20260619 08:19:18.253042 11279 master_runner.cc:305] Clearing existing system tablet
I20260619 08:19:18.253168 11279 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:18.253228 11279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:18.253980 11279 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:18.254482 11378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:18.254606 11279 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:18.254666 11279 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:18.254709 11279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/wal
uuid: "bed09c04d0564fecbde79ff3e9cfe92f"
format_stamp: "Formatted at 2026-06-19 08:19:18 on dist-test-slave-d3n8"
server_key: "1326f3adc89d49190c46023d36acd2b3"
server_key_iv: "9f05a61dd254d82256d915618fa055f1"
server_key_version: "encryptionkey@0"
I20260619 08:19:18.255299 11279 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:18.256714 11279 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 08:19:18.256785 11279 log.cc:1196] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 08:19:18.257018 11279 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f: Deleting consensus metadata
I20260619 08:19:18.257148 11279 master_runner.cc:315] Copying system tablet from 127.4.151.190:44597
I20260619 08:19:18.257771 11279 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f: tablet copy: Beginning tablet copy session from remote peer at address 127.4.151.190:44597
I20260619 08:19:18.264353 11384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.258080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.0.0.1:33388)
0619 08:19:18.258186 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:18.258196 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:18.258262 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:18.258401 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:18.258401 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:18.258567 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:18.258622 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.258624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.259569 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.259569 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.260131 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.260133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.260590 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.260591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.260637 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.261128 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:18.261133 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:18.261478 (+ 345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:18.263338 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.263338 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.263339 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.263529 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.263734 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.263734 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.263735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.263804 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.264088 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:18.264089 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:18.264130 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 08:19:18.264186 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 08:19:18.264273 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":67,"threads_started":1}
I20260619 08:19:18.264405 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.258157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33388 (local address 127.4.151.190:44597)
0619 08:19:18.258207 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.258209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.258244 (+ 35us) server_negotiation.cc:407] Connection header received
0619 08:19:18.258312 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.258313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.258323 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.258399 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:18.258691 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.259487 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.260210 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.260516 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.260588 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.261588 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:18.261589 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:18.261590 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:18.261598 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:18.263243 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.263607 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.263608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.263608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.263668 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.263882 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.263882 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.263883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.264023 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:18.264090 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:18.264234 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 08:19:18.264312 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 08:19:18.264712 10835 tablet_copy_service.cc:140] P d6dd1d19eb084efbbf3cca9f38d54286: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bed09c04d0564fecbde79ff3e9cfe92f ({username='oryx', principal='oryx/127.4.151.189@KRBTEST.COM'} at 127.0.0.1:33388)
I20260619 08:19:18.264810 10835 tablet_copy_service.cc:161] P d6dd1d19eb084efbbf3cca9f38d54286: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bed09c04d0564fecbde79ff3e9cfe92f at {username='oryx', principal='oryx/127.4.151.189@KRBTEST.COM'} at 127.0.0.1:33388: session id = bed09c04d0564fecbde79ff3e9cfe92f-00000000000000000000000000000000
I20260619 08:19:18.265601 10835 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 08:19:18.266243 11279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 08:19:18.267505 11279 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f: tablet copy: Starting download of 0 data blocks...
I20260619 08:19:18.267652 11279 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f: tablet copy: Starting download of 1 WAL segments...
I20260619 08:19:18.268579 11279 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 08:19:18.269490 10835 tablet_copy_service.cc:342] P d6dd1d19eb084efbbf3cca9f38d54286: Request end of tablet copy session bed09c04d0564fecbde79ff3e9cfe92f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.151.189@KRBTEST.COM'} at 127.0.0.1:33388
I20260619 08:19:18.269560 10835 tablet_copy_service.cc:433] P d6dd1d19eb084efbbf3cca9f38d54286: ending tablet copy session bed09c04d0564fecbde79ff3e9cfe92f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bed09c04d0564fecbde79ff3e9cfe92f
I20260619 08:19:18.271879 11279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:18.273152 11385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:18.273195 11386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:18.274502 11388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:18.274621 11279 server_base.cc:1034] running on GCE node
I20260619 08:19:18.274750 11279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:18.274909 11279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:18.276058 11279 hybrid_clock.cc:648] HybridClock initialized: now 1781857158276051 us; error 24 us; skew 500 ppm
Jun 19 08:19:18 dist-test-slave-d3n8 krb5kdc[10770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857158, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:18.277531 11279 init.cc:377] Logged in from keytab as oryx/127.4.151.189@KRBTEST.COM (short username oryx)
I20260619 08:19:18.277820 11279 webserver.cc:460] Webserver started at http://127.4.151.189:39675/ using document root <none> and password file <none>
I20260619 08:19:18.277945 11279 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:18.277981 11279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:18.278638 11279 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:19:18.279165 11395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:18.279285 11279 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:19:18.279343 11279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/wal
uuid: "bed09c04d0564fecbde79ff3e9cfe92f"
format_stamp: "Formatted at 2026-06-19 08:19:18 on dist-test-slave-d3n8"
server_key: "1326f3adc89d49190c46023d36acd2b3"
server_key_iv: "9f05a61dd254d82256d915618fa055f1"
server_key_version: "encryptionkey@0"
I20260619 08:19:18.279402 11279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:18.292769 11279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:18.294459 11279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:18.297544 11279 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.189:44003
I20260619 08:19:18.297945 11279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-1/data/info.pb
I20260619 08:19:18.298151 11447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.189:44003 every 8 connection(s)
I20260619 08:19:18.299820 11448 sys_catalog.cc:263] Verifying existing consensus state
I20260619 08:19:18.301031 11448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f: Bootstrap starting.
I20260619 08:19:18.304136 11448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:19:18.304337 11448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f: Bootstrap complete.
I20260619 08:19:18.304533 11448 raft_consensus.cc:348] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: NON_VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: true } }
I20260619 08:19:18.304636 11448 raft_consensus.cc:729] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bed09c04d0564fecbde79ff3e9cfe92f, State: Initialized, Role: LEARNER
I20260619 08:19:18.304675 11448 consensus_queue.cc:260] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: NON_VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: true } }
I20260619 08:19:18.304926 11448 sys_catalog.cc:564] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:19:18.304940 11452 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: NON_VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: true } } }
I20260619 08:19:18.305151 11452 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [sys.catalog]: This master's current role is: LEARNER
I20260619 08:19:18.307129 11463 catalog_manager.cc:1259] Loaded cluster ID: edb1d5b47cc64477b0439e888897e14a
I20260619 08:19:18.307183 11463 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f: loading cluster ID for follower catalog manager: success
I20260619 08:19:18.307317 11449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.298264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51370 (local address 127.4.151.189:44003)
0619 08:19:18.298450 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.298453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.298462 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:19:18.298492 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.298493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.298505 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.299262 (+ 757us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:18.299300 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.300323 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.301356 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.301730 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.302049 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.303262 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:18.303263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:18.303264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:18.303270 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:18.305092 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.305785 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.305786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.305786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.305842 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.306101 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.306102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.306102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.306281 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:18.307090 (+ 809us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:18.307125 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 08:19:18.307219 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":92,"threads_started":1}
I20260619 08:19:18.308233 11463 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f: acquiring CA information for follower catalog manager: success
I20260619 08:19:18.308542 11463 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for oryx/127.4.151.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.151.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.151.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.188.keytab.
Entry for principal oryx/127.4.151.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.188.keytab.
I20260619 08:19:18.337028 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.188:46337
--webserver_interface=127.4.151.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.188.keytab
--principal=oryx/127.4.151.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:40861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:44597,127.4.151.189:44003,127.4.151.188:46337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:18.423027 11466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:18.423243 11466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:18.423266 11466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:18.424361 11466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:19:18.424398 11466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:18.424412 11466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:18.424425 11466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:19:18.424437 11466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:19:18.425969 11466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:40861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:44597,127.4.151.189:44003,127.4.151.188:46337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.188.keytab
--principal=oryx/127.4.151.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.188:46337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.151.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:18.426183 11466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:18.426357 11466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:18.431504 11471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:18.431511 11472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:18.431519 11474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:18.431766 11466 server_base.cc:1034] running on GCE node
I20260619 08:19:18.432420 11466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:18.432797 11466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:18.434087 11466 hybrid_clock.cc:648] HybridClock initialized: now 1781857158434090 us; error 46 us; skew 500 ppm
Jun 19 08:19:18 dist-test-slave-d3n8 krb5kdc[10770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857158, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:18.437044 11466 init.cc:377] Logged in from keytab as oryx/127.4.151.188@KRBTEST.COM (short username oryx)
I20260619 08:19:18.437671 11466 webserver.cc:460] Webserver started at http://127.4.151.188:41759/ using document root <none> and password file <none>
I20260619 08:19:18.437969 11466 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:18.438050 11466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:18.438215 11466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:18.439574 11466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/data/instance:
uuid: "4bb90fb221c1411a84e6f320351deacc"
format_stamp: "Formatted at 2026-06-19 08:19:18 on dist-test-slave-d3n8"
server_key: "05cb428921ea4f42e09e89476c53519f"
server_key_iv: "9291a74635d417f0b43ebd8f0c1a764c"
server_key_version: "encryptionkey@0"
I20260619 08:19:18.440013 11466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/wal/instance:
uuid: "4bb90fb221c1411a84e6f320351deacc"
format_stamp: "Formatted at 2026-06-19 08:19:18 on dist-test-slave-d3n8"
server_key: "05cb428921ea4f42e09e89476c53519f"
server_key_iv: "9291a74635d417f0b43ebd8f0c1a764c"
server_key_version: "encryptionkey@0"
I20260619 08:19:18.441731 11466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260619 08:19:18.442499 11481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:18.442682 11466 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:19:18.442780 11466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/wal
uuid: "4bb90fb221c1411a84e6f320351deacc"
format_stamp: "Formatted at 2026-06-19 08:19:18 on dist-test-slave-d3n8"
server_key: "05cb428921ea4f42e09e89476c53519f"
server_key_iv: "9291a74635d417f0b43ebd8f0c1a764c"
server_key_version: "encryptionkey@0"
I20260619 08:19:18.442865 11466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:18.478847 11466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:18.481454 11466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:18.481611 11466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:18.485313 11466 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.188:46337
I20260619 08:19:18.485347 11533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.188:46337 every 8 connection(s)
I20260619 08:19:18.485754 11466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/data/info.pb
I20260619 08:19:18.486768 11534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:18.488296 11534 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } has no permanent_uuid. Determining permanent_uuid...
I20260619 08:19:18.493106 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 11466
Jun 19 08:19:18 dist-test-slave-d3n8 krb5kdc[10770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857158, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.188@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM
I20260619 08:19:18.497785 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.489021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33402 (local address 127.4.151.190:44597)
0619 08:19:18.489102 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.489112 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.489238 (+ 126us) server_negotiation.cc:407] Connection header received
0619 08:19:18.489774 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.489776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.489786 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.489848 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:18.490472 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.491044 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.491987 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.492172 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.492229 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.494465 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:18.494466 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:18.494467 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:18.494475 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:18.496263 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.496677 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.496678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.496678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.496735 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.497020 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.497020 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.497021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.497170 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:18.497243 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:18.497572 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 08:19:18.497676 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 08:19:18.497879 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.488925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.0.0.1:33402)
0619 08:19:18.489104 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:18.489145 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:18.489662 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:18.489922 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:18.489926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:18.490205 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:18.490393 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.490397 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.491123 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.491124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.491889 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.491891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.492257 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.492258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.492359 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.492831 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:18.492838 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:18.494375 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:18.496348 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.496353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.496355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.496604 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.496794 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.496794 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.496794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.496914 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.497240 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:18.497244 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:18.497420 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 08:19:18.497536 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 08:19:18.497684 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":109,"threads_started":1}
I20260619 08:19:18.498551 11534 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 08:19:18 dist-test-slave-d3n8 krb5kdc[10770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.151.188@KRBTEST.COM
Jun 19 08:19:18 dist-test-slave-d3n8 krb5kdc[10770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857158, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.188@KRBTEST.COM for oryx/127.4.151.189@KRBTEST.COM
I20260619 08:19:18.503172 11537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.493637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41468 (local address 127.4.151.188:46337)
0619 08:19:18.493785 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.493790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.493814 (+ 24us) server_negotiation.cc:407] Connection header received
0619 08:19:18.493831 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.493832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.493847 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.493953 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:18.494606 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.495563 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.496599 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.497034 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.497083 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.499682 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:18.499688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:18.499689 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:18.499695 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:18.501973 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.502347 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.502350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.502351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.502415 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.502612 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.502613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.502613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.502785 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:18.502940 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:18.503038 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 08:19:18.503101 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
I20260619 08:19:18.505548 11449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.498902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51382 (local address 127.4.151.189:44003)
0619 08:19:18.498949 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.498951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.498959 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:19:18.498985 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.498986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.498999 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.499044 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:18.499304 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.499809 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.500712 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.500982 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.501072 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.503007 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:18.503008 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:18.503009 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:18.503017 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:18.504432 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.504786 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.504788 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.504788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.504845 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.505022 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.505023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.505023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.505135 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:18.505210 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:18.505349 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 08:19:18.505452 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 08:19:18.505550 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.498832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:44003 (local address 127.0.0.1:51382)
0619 08:19:18.498875 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:18.498884 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:18.498915 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:18.499056 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:18.499057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:18.499182 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:18.499222 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.499223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.499915 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.499916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.500573 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.500575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.501070 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.501071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.501171 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.501550 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:18.501553 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:18.502921 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:18.504531 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.504532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.504532 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.504700 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.504895 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.504896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.504897 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.504949 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.505201 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:18.505202 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:18.505243 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 08:19:18.505326 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 08:19:18.505443 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 08:19:18.505924 11534 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 08:19:18 dist-test-slave-d3n8 krb5kdc[10770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857158, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.188@KRBTEST.COM for oryx/127.4.151.188@KRBTEST.COM
I20260619 08:19:18.512456 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.506283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:46337 (local address 127.0.0.1:41470)
0619 08:19:18.506373 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:18.506383 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:18.506430 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:18.506554 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:18.506554 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:18.506706 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:18.506738 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.506740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.507347 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.507348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.507809 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.507810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.508194 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.508194 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.508240 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.508586 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:18.508590 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:18.509893 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:18.511436 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.511436 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.511437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.511627 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.511820 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.511821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.511821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.511871 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.512220 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:18.512221 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:18.512257 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 08:19:18.512313 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 08:19:18.512390 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 08:19:18.512521 11537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.506339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41470 (local address 127.4.151.188:46337)
0619 08:19:18.506375 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.506377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.506416 (+ 39us) server_negotiation.cc:407] Connection header received
0619 08:19:18.506482 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.506482 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.506491 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.506549 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:18.506800 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.507252 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.507900 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.508112 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.508195 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.510024 (+ 1829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:18.510026 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:18.510026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:18.510032 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:18.511338 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.511704 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.511705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.511705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.511757 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.511961 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.511962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.511962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.512149 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:18.512214 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:18.512347 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 08:19:18.512439 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":16}
I20260619 08:19:18.514158 11534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc: Bootstrap starting.
I20260619 08:19:18.514976 11534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:18.515257 11534 log.cc:824] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:18.516170 11534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc: No bootstrap required, opened a new log
I20260619 08:19:18.517761 11534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } }
I20260619 08:19:18.517884 11534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:18.517906 11534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bb90fb221c1411a84e6f320351deacc, State: Initialized, Role: FOLLOWER
I20260619 08:19:18.518065 11534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } }
I20260619 08:19:18.518384 11534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:19:18.518360 11539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } } }
I20260619 08:19:18.518456 11539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [sys.catalog]: This master's current role is: FOLLOWER
W20260619 08:19:18.520105 11550 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:19:18.520164 11550 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:19:18.522447 11466 master_runner.cc:428] Detected that this master 4bb90fb221c1411a84e6f320351deacc is joining an existing cluster
I20260619 08:19:18.522507 11466 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.151.188:46337
I20260619 08:19:18.554502 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 10877
I20260619 08:19:18.558195 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.523703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33408 (local address 127.4.151.190:44597)
0619 08:19:18.523807 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.523809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.524073 (+ 264us) server_negotiation.cc:407] Connection header received
0619 08:19:18.524280 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.524282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.524295 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.524329 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:18.524787 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.525585 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.526914 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.527239 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.527283 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.554252 (+ 26969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:18.554254 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:18.554254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:18.554263 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:18.556587 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.557209 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.557210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.557210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.557278 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.557522 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.557523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.557524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.557680 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:18.557723 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:18.557942 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 08:19:18.558048 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 08:19:18.558602 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.523886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.0.0.1:33408)
0619 08:19:18.524015 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:18.524025 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:18.524099 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:18.524400 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:18.524401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:18.524626 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:18.524694 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.524695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.525679 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.525680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.526807 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.526809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.527694 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.527695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.527765 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.528220 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:18.528225 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:18.554123 (+ 25898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:18.556850 (+ 2727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.556852 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.556852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.557110 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.557349 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.557349 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.557350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.557455 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.557767 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:18.557767 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:18.557832 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 08:19:18.558317 (+ 485us) client_negotiation.cc:240] Negotiation successful
0619 08:19:18.558445 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260619 08:19:18.559093 11449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.523762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51388 (local address 127.4.151.189:44003)
0619 08:19:18.523841 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.523843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.523941 (+ 98us) server_negotiation.cc:407] Connection header received
0619 08:19:18.524065 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.524066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.524078 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.524109 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:18.524556 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.525300 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.527702 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.528028 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.528073 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.554244 (+ 26171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:18.554247 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:18.554247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:18.554257 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:18.557879 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.558299 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.558301 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.558301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.558358 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.558562 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.558563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.558563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.558688 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:18.558730 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:18.558920 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 08:19:18.559002 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 08:19:18.559410 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.523703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:44003 (local address 127.0.0.1:51388)
0619 08:19:18.523869 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:18.523881 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:18.523979 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:18.524176 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:18.524177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:18.524370 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:18.524453 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.524467 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.525425 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.525426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.527598 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.527601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.528149 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.528150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.528213 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.528524 (+ 311us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:18.528529 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:18.554105 (+ 25576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:18.557978 (+ 3873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.557979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.557979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.558210 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.558427 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.558428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.558428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.558493 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.558775 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:18.558775 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:18.558817 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 08:19:18.559202 (+ 385us) client_negotiation.cc:240] Negotiation successful
0619 08:19:18.559307 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":97,"threads_started":1}
I20260619 08:19:18.562533 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:41629
--local_ip_for_outbound_sockets=127.4.151.129
--tserver_master_addrs=127.4.151.190:44597
--webserver_port=38461
--webserver_interface=127.4.151.129
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.129.keytab
--principal=oryx/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:40861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.151.190:44597,127.4.151.189:44003
--tserver_master_addrs=127.4.151.190:44597,127.4.151.189:44003,127.4.151.188:46337 with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 08:19:18 dist-test-slave-d3n8 krb5kdc[10770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857157, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.190@KRBTEST.COM for oryx/127.4.151.188@KRBTEST.COM
I20260619 08:19:18.567394 11537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.524105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41486 (local address 127.4.151.188:46337)
0619 08:19:18.524138 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.524140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.524340 (+ 200us) server_negotiation.cc:407] Connection header received
0619 08:19:18.524418 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.524419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.524427 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.524452 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:18.524848 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.525583 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.526715 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.526981 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.527019 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.564042 (+ 37023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:18.564044 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:18.564044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:18.564051 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:18.565831 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.566246 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.566246 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.566247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.566296 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.566533 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.566534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.566534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.566664 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:18.566702 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:18.566866 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 08:19:18.566969 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 08:19:18.568305 11558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.524172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:46337 (local address 127.0.0.1:41486)
0619 08:19:18.524291 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:18.524300 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:18.524358 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:18.524554 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:18.524555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:18.524727 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:18.524780 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.524781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.525688 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.525688 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.526585 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.526588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.527093 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.527093 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.527156 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.527637 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:18.527643 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:18.563909 (+ 36266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:18.565922 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.565923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.565923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.566142 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.566412 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.566412 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.566413 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.566473 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.566739 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:18.566740 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:18.566774 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 08:19:18.568008 (+ 1234us) client_negotiation.cc:240] Negotiation successful
0619 08:19:18.568143 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":54,"thread_start_us":67,"threads_started":1}
I20260619 08:19:18.569851 11372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.560826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:46337 (local address 127.0.0.1:41494)
0619 08:19:18.560894 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:18.560906 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:18.560947 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:18.561225 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:18.561227 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:18.561438 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:18.561501 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.561503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.562502 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.562503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.563342 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.563345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.563856 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.563857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.563933 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.564613 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:18.564619 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:18.566582 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:18.568628 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.568629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.568629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.568843 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.569078 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.569079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.569079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.569135 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.569556 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:18.569557 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:18.569602 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 08:19:18.569652 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 08:19:18.569752 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 08:19:18.570279 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.560945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41494 (local address 127.4.151.188:46337)
0619 08:19:18.561084 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.561086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.561092 (+ 6us) server_negotiation.cc:407] Connection header received
0619 08:19:18.561127 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.561128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.561139 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.561200 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:18.561585 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.562436 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.563428 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.563753 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.565538 (+ 1785us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.566677 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:18.566678 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:18.566678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:18.566684 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:18.568526 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.568934 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.568935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.568935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.569000 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.569201 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.569202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.569202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.569313 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:18.570033 (+ 720us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:18.570073 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 08:19:18.570160 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":94,"threads_started":1}
I20260619 08:19:18.570694 10815 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.151.188:46337: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4bb90fb221c1411a84e6f320351deacc"
member_type: NON_VOTER
last_known_addr {
host: "127.4.151.188"
port: 46337
}
attrs {
promote: true
}
}
dest_uuid: "d6dd1d19eb084efbbf3cca9f38d54286"
cas_config_opid_index: 5
I20260619 08:19:18.570971 10815 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: NON_VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: true } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: NON_VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: true } }
I20260619 08:19:18.571718 11508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:18.571666 11374 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4bb90fb221c1411a84e6f320351deacc (127.4.151.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: NON_VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: true } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: NON_VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: true } } }
I20260619 08:19:18.572496 11508 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 1 FOLLOWER]: Refusing update from remote peer d6dd1d19eb084efbbf3cca9f38d54286: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 08:19:18.572747 11374 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.151.188:46337
I20260619 08:19:18.572834 11374 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: NON_VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: true } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: NON_VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: true } } }
I20260619 08:19:18.572923 11374 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:18.572974 11374 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: NON_VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:19:18.573451 11466 master.cc:537] Master@127.4.151.188:46337 shutting down...
I20260619 08:19:18.574373 11508 raft_consensus.cc:858] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: NON_VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: NON_VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: true } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: NON_VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: true } }
W20260619 08:19:18.574445 11508 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: NON_VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: NON_VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: true } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: NON_VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: true } }
I20260619 08:19:18.574918 11508 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER bed09c04d0564fecbde79ff3e9cfe92f (127.4.151.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: NON_VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: true } } }
I20260619 08:19:18.576210 11539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6dd1d19eb084efbbf3cca9f38d54286. Latest consensus state: current_term: 1 leader_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: NON_VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: true } } }
I20260619 08:19:18.576306 11539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 08:19:18.576341 11539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: NON_VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: true } } }
I20260619 08:19:18.576380 11539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 08:19:18.576406 11539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: NON_VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: true } } }
I20260619 08:19:18.576437 11539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 08:19:18.577986 11562 mvcc.cc:205] Tried to move back new op lower bound from 7298486917081350144 to 7298486916974477312. Current Snapshot: MvccSnapshot[applied={T|T < 7298486916985819136}]
I20260619 08:19:18.579043 11374 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: NON_VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: true } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: NON_VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: true } } }
I20260619 08:19:18.579139 11374 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [sys.catalog]: This master's current role is: LEADER
W20260619 08:19:18.579392 10802 consensus_peers.cc:487] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 -> Peer 4bb90fb221c1411a84e6f320351deacc (127.4.151.188:46337): Couldn't send request to peer 4bb90fb221c1411a84e6f320351deacc. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 08:19:18.580106 11466 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260619 08:19:18.580191 11466 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260619 08:19:18.580237 11466 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc: stopping tablet replica
I20260619 08:19:18.580375 11449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.571617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51402 (local address 127.4.151.189:44003)
0619 08:19:18.571668 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.571670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.575784 (+ 4114us) server_negotiation.cc:407] Connection header received
0619 08:19:18.575814 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.575815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.575838 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.575892 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:18.576641 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.577450 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.578855 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.579487 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.579557 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.579636 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:18.580271 (+ 635us) server_negotiation.cc:299] Negotiation successful
0619 08:19:18.580312 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 08:19:18.580147 11372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.571510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:44003 (local address 127.0.0.1:51402)
0619 08:19:18.575648 (+ 4138us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:18.575661 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:18.575727 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:18.576505 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:18.576506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:18.576508 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:18.576554 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.576555 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.577535 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.577536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.578721 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.578725 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.579863 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.579864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.579941 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.579975 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 08:19:18.580032 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 08:19:18.580076 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4118}
I20260619 08:19:18.580945 11422 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 1 LEARNER]: Refusing update from remote peer d6dd1d19eb084efbbf3cca9f38d54286: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 08:19:18.581156 11561 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: NON_VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 08:19:18.581760 11422 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4bb90fb221c1411a84e6f320351deacc (127.4.151.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: NON_VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: true } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: NON_VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: true } } }
I20260619 08:19:18.583181 11452 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6dd1d19eb084efbbf3cca9f38d54286. Latest consensus state: current_term: 1 leader_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: NON_VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: true } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: NON_VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: true } } }
I20260619 08:19:18.583282 11452 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [sys.catalog]: This master's current role is: LEARNER
I20260619 08:19:18.583319 11452 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: NON_VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: true } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: NON_VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: true } } }
I20260619 08:19:18.583364 11452 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [sys.catalog]: This master's current role is: LEARNER
I20260619 08:19:18.583663 11374 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286: attempting to promote NON_VOTER bed09c04d0564fecbde79ff3e9cfe92f to VOTER
I20260619 08:19:18.583847 11374 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: NON_VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: true } }
I20260619 08:19:18.584256 11422 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 1 LEARNER]: Refusing update from remote peer d6dd1d19eb084efbbf3cca9f38d54286: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 08:19:18.584563 11571 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260619 08:19:18.584656 10802 consensus_peers.cc:487] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 -> Peer 4bb90fb221c1411a84e6f320351deacc (127.4.151.188:46337): Couldn't send request to peer 4bb90fb221c1411a84e6f320351deacc. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 08:19:18.585060 11561 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bed09c04d0564fecbde79ff3e9cfe92f (127.4.151.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: NON_VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: true } } }
I20260619 08:19:18.586032 11561 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: NON_VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: true } } }
I20260619 08:19:18.586102 11561 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:18.586426 11373 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: NON_VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: true } } }
I20260619 08:19:18.586535 11373 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:18.586874 11422 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bed09c04d0564fecbde79ff3e9cfe92f (127.4.151.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: NON_VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: true } } }
I20260619 08:19:18.587934 11570 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: NON_VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: true } } }
I20260619 08:19:18.588019 11570 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:19:18.599757 11466 master.cc:552] Master@127.4.151.188:46337 shutdown complete.
I20260619 08:19:18.599823 11466 master_runner.cc:305] Clearing existing system tablet
I20260619 08:19:18.599965 11466 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:18.600023 11466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:18.600803 11466 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:19:18.601478 11573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:18.601647 11466 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:18.601711 11466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 08:19:18.601735 11466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/wal
uuid: "4bb90fb221c1411a84e6f320351deacc"
format_stamp: "Formatted at 2026-06-19 08:19:18 on dist-test-slave-d3n8"
server_key: "05cb428921ea4f42e09e89476c53519f"
server_key_iv: "9291a74635d417f0b43ebd8f0c1a764c"
server_key_version: "encryptionkey@0"
I20260619 08:19:18.602511 11466 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:18.604266 11466 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 08:19:18.604341 11466 log.cc:1196] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 08:19:18.604609 11466 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc: Deleting consensus metadata
I20260619 08:19:18.604740 11466 master_runner.cc:315] Copying system tablet from 127.4.151.190:44597
I20260619 08:19:18.605397 11466 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc: tablet copy: Beginning tablet copy session from remote peer at address 127.4.151.190:44597
I20260619 08:19:18.615406 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.605824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33416 (local address 127.4.151.190:44597)
0619 08:19:18.605879 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.605881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.608248 (+ 2367us) server_negotiation.cc:407] Connection header received
0619 08:19:18.608309 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.608310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.608324 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.608394 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:18.608791 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.609683 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.610769 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.611091 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.611161 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.612308 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:18.612309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:18.612309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:18.612317 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:18.614122 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.614551 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.614552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.614553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.614614 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.614851 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.614852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.614852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.614983 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:18.615059 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:18.615232 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 08:19:18.615311 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 08:19:18.615411 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.605728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.0.0.1:33416)
0619 08:19:18.608138 (+ 2410us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:18.608151 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:18.608239 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:18.608416 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:18.608417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:18.608630 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:18.608698 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.608701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.609785 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.609786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.610671 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.610674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.611173 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.611174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.611249 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.611788 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:18.611793 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:18.612204 (+ 411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:18.614223 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.614224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.614225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.614467 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.614693 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.614694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.614694 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.614777 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.615064 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:18.615065 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:18.615117 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 08:19:18.615197 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 08:19:18.615307 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2386,"thread_start_us":62,"threads_started":1}
I20260619 08:19:18.616182 10835 tablet_copy_service.cc:140] P d6dd1d19eb084efbbf3cca9f38d54286: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4bb90fb221c1411a84e6f320351deacc ({username='oryx', principal='oryx/127.4.151.188@KRBTEST.COM'} at 127.0.0.1:33416)
I20260619 08:19:18.616251 10835 tablet_copy_service.cc:161] P d6dd1d19eb084efbbf3cca9f38d54286: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4bb90fb221c1411a84e6f320351deacc at {username='oryx', principal='oryx/127.4.151.188@KRBTEST.COM'} at 127.0.0.1:33416: session id = 4bb90fb221c1411a84e6f320351deacc-00000000000000000000000000000000
I20260619 08:19:18.616830 10835 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 08:19:18.617406 11466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 08:19:18.618886 11466 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc: tablet copy: Starting download of 0 data blocks...
I20260619 08:19:18.619175 11466 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc: tablet copy: Starting download of 1 WAL segments...
I20260619 08:19:18.620244 11466 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 08:19:18.621237 10835 tablet_copy_service.cc:342] P d6dd1d19eb084efbbf3cca9f38d54286: Request end of tablet copy session 4bb90fb221c1411a84e6f320351deacc-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.151.188@KRBTEST.COM'} at 127.0.0.1:33416
I20260619 08:19:18.621301 10835 tablet_copy_service.cc:433] P d6dd1d19eb084efbbf3cca9f38d54286: ending tablet copy session 4bb90fb221c1411a84e6f320351deacc-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4bb90fb221c1411a84e6f320351deacc
I20260619 08:19:18.624404 11466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:18.625906 11580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:18.626525 11581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:18.626783 11583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:18.627166 11466 server_base.cc:1034] running on GCE node
I20260619 08:19:18.627261 11466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:18.627393 11466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:18.628528 11466 hybrid_clock.cc:648] HybridClock initialized: now 1781857158628517 us; error 24 us; skew 500 ppm
Jun 19 08:19:18 dist-test-slave-d3n8 krb5kdc[10770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857158, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:18.630347 11466 init.cc:377] Logged in from keytab as oryx/127.4.151.188@KRBTEST.COM (short username oryx)
I20260619 08:19:18.630649 11466 webserver.cc:460] Webserver started at http://127.4.151.188:46717/ using document root <none> and password file <none>
I20260619 08:19:18.630766 11466 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:18.630795 11466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:18.631629 11466 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:18.632243 11590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:18.632417 11466 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:18.632477 11466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/wal
uuid: "4bb90fb221c1411a84e6f320351deacc"
format_stamp: "Formatted at 2026-06-19 08:19:18 on dist-test-slave-d3n8"
server_key: "05cb428921ea4f42e09e89476c53519f"
server_key_iv: "9291a74635d417f0b43ebd8f0c1a764c"
server_key_version: "encryptionkey@0"
I20260619 08:19:18.632534 11466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:18.660534 11466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
W20260619 08:19:18.662487 11560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:18.662770 11560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:18.662822 11560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 08:19:18.662811 11466 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260619 08:19:18.664856 11560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:18.664922 11560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:18.664973 11560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:19:18.666036 11466 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.188:46337
I20260619 08:19:18.666153 11645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.188:46337 every 8 connection(s)
I20260619 08:19:18.666373 11466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/master-2/data/info.pb
I20260619 08:19:18.667181 11560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:40861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.129.keytab
--principal=oryx/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:41629
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=38461
--tserver_master_addrs=127.4.151.190:44597,127.4.151.189:44003,127.4.151.188:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:18.667377 11646 sys_catalog.cc:263] Verifying existing consensus state
I20260619 08:19:18.667474 11560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:18.667708 11560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:19:18.667879 11646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc: Bootstrap starting.
I20260619 08:19:18.670258 11646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:19:18.670528 11646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc: Bootstrap complete.
I20260619 08:19:18.670732 11646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: NON_VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: true } }
I20260619 08:19:18.670827 11646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4bb90fb221c1411a84e6f320351deacc, State: Initialized, Role: LEARNER
I20260619 08:19:18.670869 11646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: NON_VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: true } }
I20260619 08:19:18.671060 11646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:19:18.671066 11654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: NON_VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: true } } }
I20260619 08:19:18.671280 11654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [sys.catalog]: This master's current role is: LEARNER
I20260619 08:19:18.673832 11665 catalog_manager.cc:1259] Loaded cluster ID: edb1d5b47cc64477b0439e888897e14a
W20260619 08:19:18.673792 11653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:18.673889 11665 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc: loading cluster ID for follower catalog manager: success
W20260619 08:19:18.673822 11649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:18.673995 11651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:18.675098 11665 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc: acquiring CA information for follower catalog manager: success
I20260619 08:19:18.675282 11560 server_base.cc:1034] running on GCE node
I20260619 08:19:18.675438 11665 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 08:19:18.675514 11560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:18.675786 11560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:18.677105 11560 hybrid_clock.cc:648] HybridClock initialized: now 1781857158677083 us; error 37 us; skew 500 ppm
Jun 19 08:19:18 dist-test-slave-d3n8 krb5kdc[10770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857158, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:18.680297 11560 init.cc:377] Logged in from keytab as oryx/127.4.151.129@KRBTEST.COM (short username oryx)
I20260619 08:19:18.680800 11560 webserver.cc:460] Webserver started at http://127.4.151.129:38461/ using document root <none> and password file <none>
I20260619 08:19:18.681056 11560 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:18.681125 11560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:18.682857 11560 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:18.683588 11672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:18.683828 11560 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:18.683919 11560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "130acc74148042b68fad3073fe31193a"
format_stamp: "Formatted at 2026-06-19 08:19:17 on dist-test-slave-d3n8"
server_key: "01fb153b3119cd54434167d1e123c2cf"
server_key_iv: "b5018a61aff0b78a3278b1f14ac48864"
server_key_version: "encryptionkey@0"
I20260619 08:19:18.684361 11560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:18.729977 11560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:18.732427 11560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:18.732549 11560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:18.732805 11560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:18.733131 11560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:18.733165 11560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:18.733187 11560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:18.733233 11560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:18.739409 11560 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:41629
I20260619 08:19:18.739463 11785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:41629 every 8 connection(s)
I20260619 08:19:18.739804 11560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
I20260619 08:19:18.741340 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 11560
I20260619 08:19:18.741569 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 11011
Jun 19 08:19:18 dist-test-slave-d3n8 krb5kdc[10770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857158, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.188@KRBTEST.COM
I20260619 08:19:18.748098 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:45771
--local_ip_for_outbound_sockets=127.4.151.130
--tserver_master_addrs=127.4.151.190:44597
--webserver_port=43689
--webserver_interface=127.4.151.130
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.130.keytab
--principal=oryx/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:40861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.151.190:44597,127.4.151.189:44003
--tserver_master_addrs=127.4.151.190:44597,127.4.151.189:44003,127.4.151.188:46337 with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 08:19:18 dist-test-slave-d3n8 krb5kdc[10770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857158, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM
Jun 19 08:19:18 dist-test-slave-d3n8 krb5kdc[10770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857158, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.189@KRBTEST.COM
I20260619 08:19:18.751936 11790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.740667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:60101 (local address 127.4.151.188:46337)
0619 08:19:18.740825 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.740827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.741011 (+ 184us) server_negotiation.cc:407] Connection header received
0619 08:19:18.742457 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.742458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.742473 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.742513 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:18.743672 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.744437 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.745550 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.745779 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.745868 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.748472 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:18.748474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:18.748474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:18.748482 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:18.750406 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.750807 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.750808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.750809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.750887 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.751192 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.751193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.751194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.751340 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:18.751385 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:18.751747 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 08:19:18.751837 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":105,"threads_started":1}
I20260619 08:19:18.752139 11792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.740688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:46337 (local address 127.4.151.129:60101)
0619 08:19:18.740889 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:18.740933 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:18.742342 (+ 1409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:18.742708 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:18.742709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:18.743237 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:18.743564 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.743569 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.744547 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.744547 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.745450 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.745453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.745885 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.745886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.746008 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.746540 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:18.746548 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:18.748340 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:18.750483 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.750491 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.750493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.750722 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.750978 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.750979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.750979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.751111 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.751432 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:18.751439 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:18.751606 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 08:19:18.751715 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 08:19:18.751886 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":26,"thread_start_us":114,"threads_started":1}
I20260619 08:19:18.753551 11786 heartbeater.cc:347] Connected to a master server at 127.4.151.188:46337
I20260619 08:19:18.753660 11786 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:18.753881 11786 heartbeater.cc:511] Master 127.4.151.188:46337 requested a full tablet report, sending...
I20260619 08:19:18.754465 11610 ts_manager.cc:194] Registered new tserver with Master: 130acc74148042b68fad3073fe31193a (127.4.151.129:41629)
I20260619 08:19:18.754684 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.740997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.129:49885)
0619 08:19:18.741166 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:18.741176 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:18.742409 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:18.742701 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:18.742705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:18.743333 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:18.743564 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.743569 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.746649 (+ 3080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.746650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.747569 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.747571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.748302 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.748303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.748380 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.748797 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:18.748802 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:18.751297 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:18.753509 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.753510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.753511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.753728 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.753980 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.753981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.753981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.754032 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.754347 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:18.754348 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:18.754390 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 08:19:18.754459 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 08:19:18.754562 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":5,"thread_start_us":65,"threads_started":1}
I20260619 08:19:18.755012 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.741080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:49885 (local address 127.4.151.190:44597)
0619 08:19:18.741137 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.741139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.741249 (+ 110us) server_negotiation.cc:407] Connection header received
0619 08:19:18.742499 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.742500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.742511 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.742542 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:18.745809 (+ 3267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.746555 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.747880 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.748188 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.748225 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.751402 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:18.751404 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:18.751404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:18.751412 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:18.753434 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.753820 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.753822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.753822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.753910 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.754113 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.754114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.754114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.754258 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:18.754323 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:18.754818 (+ 495us) server_negotiation.cc:299] Negotiation successful
0619 08:19:18.754919 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 08:19:18.755141 11449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.740687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:39765 (local address 127.4.151.189:44003)
0619 08:19:18.740739 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.740741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.741035 (+ 294us) server_negotiation.cc:407] Connection header received
0619 08:19:18.742457 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.742458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.742473 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.742508 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:18.743802 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.744627 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.747851 (+ 3224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.748142 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.748204 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.751415 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:18.751417 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:18.751417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:18.751427 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:18.753794 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.754225 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.754226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.754226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.754276 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.754482 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.754482 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.754483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.754617 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:18.754667 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:18.754980 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 08:19:18.755053 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 08:19:18.755491 11787 heartbeater.cc:347] Connected to a master server at 127.4.151.190:44597
I20260619 08:19:18.755568 11787 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:18.755645 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.740688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:44003 (local address 127.4.151.129:39765)
0619 08:19:18.740894 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:18.740933 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:18.742328 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:18.742701 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:18.742705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:18.743658 (+ 953us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:18.743720 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.743722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.744722 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.744722 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.747755 (+ 3033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.747756 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.748548 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.748549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.748615 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.748999 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:18.749004 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:18.751303 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:18.753940 (+ 2637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.753941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.753941 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.754147 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.754344 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.754344 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.754345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.754406 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.754827 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:18.754827 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:18.754863 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 08:19:18.755393 (+ 530us) client_negotiation.cc:240] Negotiation successful
0619 08:19:18.755479 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
I20260619 08:19:18.755661 11787 heartbeater.cc:511] Master 127.4.151.190:44597 requested a full tablet report, sending...
I20260619 08:19:18.755944 10815 ts_manager.cc:194] Re-registered known tserver with Master: 130acc74148042b68fad3073fe31193a (127.4.151.129:41629)
I20260619 08:19:18.756047 11788 heartbeater.cc:347] Connected to a master server at 127.4.151.189:44003
I20260619 08:19:18.756099 11788 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:18.756175 11788 heartbeater.cc:511] Master 127.4.151.189:44003 requested a full tablet report, sending...
I20260619 08:19:18.756435 10815 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.151.129@KRBTEST.COM'} at 127.4.151.129:49885
I20260619 08:19:18.756551 11412 ts_manager.cc:194] Registered new tserver with Master: 130acc74148042b68fad3073fe31193a (127.4.151.129:41629)
W20260619 08:19:18.832394 11794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:18.832584 11794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:18.832607 11794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:18.834003 11794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:18.834085 11794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:18.834112 11794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:19:18.835531 11794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:40861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.130.keytab
--principal=oryx/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:45771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=43689
--tserver_master_addrs=127.4.151.190:44597,127.4.151.189:44003,127.4.151.188:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:18.835711 11794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:18.835878 11794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:18.841156 11799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:18.841194 11800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:18.841073 11802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:18.841346 11794 server_base.cc:1034] running on GCE node
I20260619 08:19:18.841621 11794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:18.841904 11794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:18.843256 11794 hybrid_clock.cc:648] HybridClock initialized: now 1781857158843240 us; error 25 us; skew 500 ppm
Jun 19 08:19:18 dist-test-slave-d3n8 krb5kdc[10770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857158, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:18.846096 11794 init.cc:377] Logged in from keytab as oryx/127.4.151.130@KRBTEST.COM (short username oryx)
I20260619 08:19:18.846618 11794 webserver.cc:460] Webserver started at http://127.4.151.130:43689/ using document root <none> and password file <none>
I20260619 08:19:18.846961 11794 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:18.847064 11794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:18.848553 11794 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:19:18.849260 11809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:18.849496 11794 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:18.849602 11794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "9817b0faddf74855b5625fa520c160d7"
format_stamp: "Formatted at 2026-06-19 08:19:17 on dist-test-slave-d3n8"
server_key: "eeaa5556caf381b3699a908939e5aad3"
server_key_iv: "5c23de858e3e0402fea84b103544de51"
server_key_version: "encryptionkey@0"
I20260619 08:19:18.850034 11794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:18.857210 11794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:18.859212 11794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:18.859387 11794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:18.859606 11794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:18.859907 11794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:18.859961 11794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:18.859999 11794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:18.860033 11794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:18.866706 11794 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:45771
I20260619 08:19:18.866745 11922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:45771 every 8 connection(s)
I20260619 08:19:18.867137 11794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
Jun 19 08:19:18 dist-test-slave-d3n8 krb5kdc[10770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857158, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM
I20260619 08:19:18.874222 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 11794
I20260619 08:19:18.874383 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 11145
Jun 19 08:19:18 dist-test-slave-d3n8 krb5kdc[10770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857158, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.188@KRBTEST.COM
Jun 19 08:19:18 dist-test-slave-d3n8 krb5kdc[10770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857158, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.189@KRBTEST.COM
I20260619 08:19:18.878561 11927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.867896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.130:51835)
0619 08:19:18.868063 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:18.868105 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:18.868671 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:18.868921 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:18.868922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:18.869451 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:18.869657 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.869664 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.870377 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.870377 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.871494 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.871496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.871893 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.871894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.872033 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.872628 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:18.872639 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:18.874504 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:18.876602 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.876610 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.876612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.876915 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.877192 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.877193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.877193 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.877339 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.877745 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:18.877752 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:18.877951 (+ 199us) client_negotiation.cc:769] Sending connection context
0619 08:19:18.878083 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 08:19:18.878281 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":53,"thread_start_us":84,"threads_started":1}
I20260619 08:19:18.878928 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.867906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:51835 (local address 127.4.151.190:44597)
0619 08:19:18.867957 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.867960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.868184 (+ 224us) server_negotiation.cc:407] Connection header received
0619 08:19:18.868807 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.868808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.868819 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.868867 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:18.869760 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.870266 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.871586 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.871801 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.871884 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.874634 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:18.874636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:18.874637 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:18.874645 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:18.876500 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.877039 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.877042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.877043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.877111 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.877443 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.877444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.877444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.877612 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:18.877653 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:18.878732 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0619 08:19:18.878824 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:19:18.880079 11449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.867887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:48107 (local address 127.4.151.189:44003)
0619 08:19:18.867937 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.867939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.868201 (+ 262us) server_negotiation.cc:407] Connection header received
0619 08:19:18.868760 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.868761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.868774 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.868824 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:18.870371 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.871122 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.872150 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.872383 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.872460 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.876497 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:18.876498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:18.876498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:18.876507 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:18.878579 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.879342 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.879343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.879343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.879409 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.879618 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.879619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.879619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.879746 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:18.879803 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:18.879948 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 08:19:18.880009 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 08:19:18.880169 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.867895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:44003 (local address 127.4.151.130:48107)
0619 08:19:18.868125 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:18.868136 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:18.868648 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:18.868891 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:18.868895 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:18.869434 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:18.869657 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.869662 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.871208 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.871209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.872074 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.872075 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.872472 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.872473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.872555 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.872964 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:18.872968 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:18.876381 (+ 3413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:18.878810 (+ 2429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.878811 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.878812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.879250 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.879481 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.879481 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.879482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.879548 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.879812 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:18.879813 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:18.879860 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 08:19:18.879933 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 08:19:18.880055 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
I20260619 08:19:18.880601 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:39947
--local_ip_for_outbound_sockets=127.4.151.131
--tserver_master_addrs=127.4.151.190:44597
--webserver_port=45983
--webserver_interface=127.4.151.131
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.131.keytab
--principal=oryx/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:40861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.151.190:44597,127.4.151.189:44003
--tserver_master_addrs=127.4.151.190:44597,127.4.151.189:44003,127.4.151.188:46337 with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:19:18.880812 11925 heartbeater.cc:347] Connected to a master server at 127.4.151.189:44003
I20260619 08:19:18.880923 11925 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:18.881155 11925 heartbeater.cc:511] Master 127.4.151.189:44003 requested a full tablet report, sending...
I20260619 08:19:18.881659 11412 ts_manager.cc:194] Registered new tserver with Master: 9817b0faddf74855b5625fa520c160d7 (127.4.151.130:45771)
I20260619 08:19:18.881786 11924 heartbeater.cc:347] Connected to a master server at 127.4.151.190:44597
I20260619 08:19:18.881835 11924 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:18.881948 11924 heartbeater.cc:511] Master 127.4.151.190:44597 requested a full tablet report, sending...
I20260619 08:19:18.882922 10815 ts_manager.cc:194] Re-registered known tserver with Master: 9817b0faddf74855b5625fa520c160d7 (127.4.151.130:45771)
I20260619 08:19:18.883538 10815 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.151.130@KRBTEST.COM'} at 127.4.151.130:51835
I20260619 08:19:18.883879 11790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.868262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:54789 (local address 127.4.151.188:46337)
0619 08:19:18.868316 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.868318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.868389 (+ 71us) server_negotiation.cc:407] Connection header received
0619 08:19:18.868763 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.868764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.868779 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.868812 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:18.869760 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.870261 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.871603 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.871935 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.871976 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.880084 (+ 8108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:18.880086 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:18.880086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:18.880093 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:18.882584 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.883013 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.883015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.883015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.883080 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:18.883277 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:18.883278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:18.883278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:18.883415 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:18.883483 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:18.883656 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 08:19:18.883778 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 08:19:18.885357 11929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.868178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:46337 (local address 127.4.151.130:54789)
0619 08:19:18.868336 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:18.868345 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:18.868674 (+ 329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:18.869277 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:18.869278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:18.869484 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:18.869657 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.869664 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.870369 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.870370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.871495 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:18.871497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:18.872069 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:18.872070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:18.872134 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:18.872818 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:18.872822 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:18.876063 (+ 3241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:18.882669 (+ 6606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.882670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.882671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.882926 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.883142 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:18.883143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:18.883144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:18.883207 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:18.883488 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:18.883488 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:18.883535 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 08:19:18.883609 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 08:19:18.883776 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":37,"thread_start_us":55,"threads_started":1}
I20260619 08:19:18.885798 11923 heartbeater.cc:347] Connected to a master server at 127.4.151.188:46337
I20260619 08:19:18.885872 11923 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:18.885951 11923 heartbeater.cc:511] Master 127.4.151.188:46337 requested a full tablet report, sending...
I20260619 08:19:18.886323 11610 ts_manager.cc:194] Registered new tserver with Master: 9817b0faddf74855b5625fa520c160d7 (127.4.151.130:45771)
W20260619 08:19:18.963624 11930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:18.963824 11930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:18.963887 11930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:18.965183 11930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:18.965250 11930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:18.965284 11930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:19:18.966791 11930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:40861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.131.keytab
--principal=oryx/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:39947
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=45983
--tserver_master_addrs=127.4.151.190:44597,127.4.151.189:44003,127.4.151.188:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:18.966971 11930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:18.967142 11930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:18.972383 11935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:18.972354 11936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:18.972582 11938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:18.972587 11930 server_base.cc:1034] running on GCE node
I20260619 08:19:18.972832 11930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:18.973083 11930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:18.974226 11930 hybrid_clock.cc:648] HybridClock initialized: now 1781857158974205 us; error 43 us; skew 500 ppm
Jun 19 08:19:18 dist-test-slave-d3n8 krb5kdc[10770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857158, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:18.976799 11930 init.cc:377] Logged in from keytab as oryx/127.4.151.131@KRBTEST.COM (short username oryx)
I20260619 08:19:18.977264 11930 webserver.cc:460] Webserver started at http://127.4.151.131:45983/ using document root <none> and password file <none>
I20260619 08:19:18.977537 11930 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:18.977620 11930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:18.979010 11930 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:18.979631 11945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:18.979869 11930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:18.979941 11930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "f5de23594c5a401cbb3b42642de5fe2e"
format_stamp: "Formatted at 2026-06-19 08:19:17 on dist-test-slave-d3n8"
server_key: "0c10a82d585dc5f6aaad0237d8472994"
server_key_iv: "ffaa232816d2b593e8331eef4cb2002b"
server_key_version: "encryptionkey@0"
I20260619 08:19:18.980327 11930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:18.988117 11930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:18.990535 11930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:18.990682 11930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:18.990881 11930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:18.991150 11930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:18.991183 11930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:18.991206 11930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:18.991220 11930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:18.997519 11930 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:39947
I20260619 08:19:18.997589 12058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:39947 every 8 connection(s)
I20260619 08:19:18.997961 11930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
Jun 19 08:19:19 dist-test-slave-d3n8 krb5kdc[10770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857158, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.189@KRBTEST.COM
Jun 19 08:19:19 dist-test-slave-d3n8 krb5kdc[10770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857158, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.188@KRBTEST.COM
I20260619 08:19:19.006245 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 11930
Jun 19 08:19:19 dist-test-slave-d3n8 krb5kdc[10770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857158, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM
I20260619 08:19:19.008685 11449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.998633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:53279 (local address 127.4.151.189:44003)
0619 08:19:18.998766 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.998768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.998983 (+ 215us) server_negotiation.cc:407] Connection header received
0619 08:19:18.999605 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.999607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.999620 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.999686 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:19.000763 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.001310 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.002343 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.002549 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.002638 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:19.005226 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:19.005227 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:19.005228 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:19.005235 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:19.006744 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:19.007306 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:19.007307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:19.007308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:19.007379 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:19.007714 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:19.007715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:19.007716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:19.007911 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:19.007965 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:19.008439 (+ 474us) server_negotiation.cc:299] Negotiation successful
0619 08:19:19.008552 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260619 08:19:19.009320 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.998637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:44003 (local address 127.4.151.131:53279)
0619 08:19:18.998841 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:18.998886 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:18.999500 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:18.999738 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:18.999741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:19.000416 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:19.000661 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:19.000667 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.001449 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.001450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:19.002254 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:19.002256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.002647 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.002648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:19.002773 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:19.003294 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:19.003301 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:19.005122 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:19.006877 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:19.006885 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:19.006888 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:19.007216 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:19.007466 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:19.007467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:19.007468 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:19.007615 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:19.008028 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:19.008035 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:19.008255 (+ 220us) client_negotiation.cc:769] Sending connection context
0619 08:19:19.008845 (+ 590us) client_negotiation.cc:240] Negotiation successful
0619 08:19:19.009044 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":13,"thread_start_us":79,"threads_started":1}
I20260619 08:19:19.009855 12061 heartbeater.cc:347] Connected to a master server at 127.4.151.189:44003
I20260619 08:19:19.009972 12061 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:19.010174 12061 heartbeater.cc:511] Master 127.4.151.189:44003 requested a full tablet report, sending...
I20260619 08:19:19.010654 11412 ts_manager.cc:194] Registered new tserver with Master: f5de23594c5a401cbb3b42642de5fe2e (127.4.151.131:39947)
I20260619 08:19:19.011669 12064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.998920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.131:46959)
0619 08:19:18.999086 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:18.999096 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:18.999513 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:18.999819 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:18.999820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:19.000373 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:19.000661 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:19.000666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.002376 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.002377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:19.003743 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:19.003744 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.004354 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.004355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:19.004442 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:19.004848 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:19.004853 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:19.007660 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:19.010586 (+ 2926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:19.010586 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:19.010587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:19.010799 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:19.011010 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:19.011011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:19.011011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:19.011078 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:19.011349 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:19.011350 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:19.011391 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 08:19:19.011463 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 08:19:19.011560 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":59,"thread_start_us":77,"threads_started":1}
I20260619 08:19:19.011718 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.999031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:46959 (local address 127.4.151.190:44597)
0619 08:19:18.999079 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.999081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.999164 (+ 83us) server_negotiation.cc:407] Connection header received
0619 08:19:18.999736 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.999737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.999746 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:18.999813 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:19.001452 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.002280 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.003834 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.004200 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.004306 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:19.007765 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:19.007785 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:19.007786 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:19.007794 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:19.009594 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:19.010880 (+ 1286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:19.010881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:19.010882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:19.010939 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:19.011145 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:19.011146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:19.011146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:19.011260 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:19.011303 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:19.011525 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 08:19:19.011614 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 08:19:19.012112 12060 heartbeater.cc:347] Connected to a master server at 127.4.151.190:44597
I20260619 08:19:19.012181 12060 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:19.012265 12060 heartbeater.cc:511] Master 127.4.151.190:44597 requested a full tablet report, sending...
I20260619 08:19:19.012508 10815 ts_manager.cc:194] Re-registered known tserver with Master: f5de23594c5a401cbb3b42642de5fe2e (127.4.151.131:39947)
I20260619 08:19:19.013015 10815 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.151.131@KRBTEST.COM'} at 127.4.151.131:46959
I20260619 08:19:19.013820 11790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.998704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:34961 (local address 127.4.151.188:46337)
0619 08:19:18.998789 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:18.998791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:18.999187 (+ 396us) server_negotiation.cc:407] Connection header received
0619 08:19:18.999871 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:18.999873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:18.999884 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:19.000099 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:19.001461 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.002271 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.003506 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.003816 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.003903 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:19.007095 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:19.007096 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:19.007097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:19.007104 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:19.010497 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:19.012855 (+ 2358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:19.012856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:19.012856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:19.012924 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:19.013134 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:19.013135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:19.013136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:19.013276 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:19.013321 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:19.013618 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 08:19:19.013706 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 08:19:19.014190 12065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:18.998637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:46337 (local address 127.4.151.131:34961)
0619 08:19:18.999043 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:18.999098 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:18.999500 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:19.000110 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:19.000111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:19.000422 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:19.000661 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:19.000667 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.002376 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.002377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:19.003393 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:19.003396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.003899 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.003900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:19.003983 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:19.004434 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:19.004439 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:19.007000 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:19.011996 (+ 4996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:19.011997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:19.011998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:19.012770 (+ 772us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:19.012999 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:19.012999 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:19.013000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:19.013059 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:19.013422 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:19.013423 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:19.013473 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 08:19:19.013581 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 08:19:19.013692 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":35,"thread_start_us":83,"threads_started":1}
I20260619 08:19:19.014747 12059 heartbeater.cc:347] Connected to a master server at 127.4.151.188:46337
I20260619 08:19:19.014803 12059 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:19.014875 12059 heartbeater.cc:511] Master 127.4.151.188:46337 requested a full tablet report, sending...
I20260619 08:19:19.015627 11610 ts_manager.cc:194] Registered new tserver with Master: f5de23594c5a401cbb3b42642de5fe2e (127.4.151.131:39947)
I20260619 08:19:19.017928 12072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:19.007995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41498 (local address 127.4.151.188:46337)
0619 08:19:19.008230 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:19.008232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:19.008501 (+ 269us) server_negotiation.cc:407] Connection header received
0619 08:19:19.008533 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:19.008534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:19.008546 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:19.008604 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:19.010369 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.011277 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.012309 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.012595 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.014249 (+ 1654us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:19.014554 (+ 305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:19.014555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:19.014555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:19.014562 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:19.016255 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:19.016677 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:19.016678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:19.016678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:19.016734 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:19.017010 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:19.017011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:19.017011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:19.017141 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:19.017254 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:19.017393 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 08:19:19.017782 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":54,"threads_started":1}
I20260619 08:19:19.019467 12070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:19.007474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33420 (local address 127.4.151.190:44597)
0619 08:19:19.008291 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:19.008293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:19.008301 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:19:19.008400 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:19.008412 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:19.008426 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:19.008518 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:19.010025 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.011084 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.012746 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.013166 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.013227 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:19.015680 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:19.015682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:19.015682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:19.015745 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:19.017606 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:19.018400 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:19.018408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:19.018409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:19.018479 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:19.018702 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:19.018703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:19.018703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:19.018831 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:19.018951 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:19.019316 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 08:19:19.019384 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":88,"threads_started":1}
I20260619 08:19:19.021546 11449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:19.012471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51408 (local address 127.4.151.189:44003)
0619 08:19:19.012507 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:19.012509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:19.012517 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:19:19.012536 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:19.012537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:19.012546 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:19.012600 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:19.015041 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.015731 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.016793 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.017111 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.017154 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:19.018843 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:19.018844 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:19.018845 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:19.018852 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:19.020346 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:19.020697 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:19.020699 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:19.020699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:19.020755 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:19.020977 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:19.020978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:19.020979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:19.021133 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:19.021213 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:19.021348 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 08:19:19.021481 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 08:19:19.021220 10815 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:19:19.023133 10815 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:19:19.030808 11372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:19.026853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:39947 (local address 127.0.0.1:49014)
0619 08:19:19.026900 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:19.026911 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:19.026947 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:19.027322 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:19.027323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:19.027325 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:19.027372 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:19.027373 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.028698 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.028699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:19.029807 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:19.029809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.030602 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.030603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:19.030665 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:19.030670 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:19:19.030699 (+ 29us) client_negotiation.cc:240] Negotiation successful
0619 08:19:19.030737 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 08:19:19.031263 12081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:19.026928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49014 (local address 127.4.151.131:39947)
0619 08:19:19.027135 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:19.027140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:19.027150 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:19:19.027191 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:19.027194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:19.027211 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:19.027448 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:19.027506 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.028601 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.029879 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.030508 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.030979 (+ 471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:19.031076 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:19.031106 (+ 30us) server_negotiation.cc:299] Negotiation successful
0619 08:19:19.031147 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":83,"threads_started":1}
I20260619 08:19:19.031366 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:19.026868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:41629 (local address 127.0.0.1:58726)
0619 08:19:19.027360 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:19.027370 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:19.027419 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:19.027762 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:19.027763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:19.027764 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:19.027798 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:19.027799 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.028982 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.028983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:19.030146 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:19.030148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.031132 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.031133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:19.031193 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:19.031197 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:19:19.031245 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 08:19:19.031288 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":94,"threads_started":1}
I20260619 08:19:19.031675 12079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:19.026899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:45771 (local address 127.0.0.1:41420)
0619 08:19:19.027494 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:19.027503 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:19.027554 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:19.028030 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:19.028031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:19.028033 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:19.028070 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:19.028071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.029654 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.029654 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:19.030560 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:19.030561 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.031434 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.031435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:19.031509 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:19.031515 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 08:19:19.031544 (+ 29us) client_negotiation.cc:240] Negotiation successful
0619 08:19:19.031585 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":61,"threads_started":1}
I20260619 08:19:19.031951 11993 tablet_service.cc:1467] Processing CreateTablet for tablet 34926e3b893f4eb3a725d78721a8ea58 (DEFAULT_TABLE table=test-table [id=5fccb5ab74044b93aac6a72784790ea1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:19.032056 12083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:19.027132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41420 (local address 127.4.151.130:45771)
0619 08:19:19.027867 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:19.027871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:19.027880 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:19:19.027915 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:19.027917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:19.027930 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:19.028137 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:19.028194 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.029574 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.030651 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.031328 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.031840 (+ 512us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:19.031927 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:19.031951 (+ 24us) server_negotiation.cc:299] Negotiation successful
0619 08:19:19.031999 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":79,"threads_started":1}
I20260619 08:19:19.032332 11993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34926e3b893f4eb3a725d78721a8ea58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:19.032712 11857 tablet_service.cc:1467] Processing CreateTablet for tablet 34926e3b893f4eb3a725d78721a8ea58 (DEFAULT_TABLE table=test-table [id=5fccb5ab74044b93aac6a72784790ea1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:19.033108 11857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34926e3b893f4eb3a725d78721a8ea58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:19.035467 12084 tablet_bootstrap.cc:492] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e: Bootstrap starting.
I20260619 08:19:19.035674 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:19.027117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58726 (local address 127.4.151.129:41629)
0619 08:19:19.027285 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:19.027289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:19.027610 (+ 321us) server_negotiation.cc:407] Connection header received
0619 08:19:19.027652 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:19.027654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:19.027669 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:19.027750 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:19.027880 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.028889 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.030238 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.030972 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.031032 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:19.031099 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:19.035532 (+ 4433us) server_negotiation.cc:299] Negotiation successful
0619 08:19:19.035591 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
I20260619 08:19:19.036054 12085 tablet_bootstrap.cc:492] T 34926e3b893f4eb3a725d78721a8ea58 P 9817b0faddf74855b5625fa520c160d7: Bootstrap starting.
I20260619 08:19:19.036253 12084 tablet_bootstrap.cc:654] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:19.036319 11720 tablet_service.cc:1467] Processing CreateTablet for tablet 34926e3b893f4eb3a725d78721a8ea58 (DEFAULT_TABLE table=test-table [id=5fccb5ab74044b93aac6a72784790ea1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:19.036594 12084 log.cc:824] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:19.036670 11720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34926e3b893f4eb3a725d78721a8ea58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:19.036720 12085 tablet_bootstrap.cc:654] T 34926e3b893f4eb3a725d78721a8ea58 P 9817b0faddf74855b5625fa520c160d7: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:19.037030 12085 log.cc:824] T 34926e3b893f4eb3a725d78721a8ea58 P 9817b0faddf74855b5625fa520c160d7: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:19.037407 12084 tablet_bootstrap.cc:492] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e: No bootstrap required, opened a new log
I20260619 08:19:19.037468 12084 ts_tablet_manager.cc:1397] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:19:19.037797 12085 tablet_bootstrap.cc:492] T 34926e3b893f4eb3a725d78721a8ea58 P 9817b0faddf74855b5625fa520c160d7: No bootstrap required, opened a new log
I20260619 08:19:19.037879 12085 ts_tablet_manager.cc:1397] T 34926e3b893f4eb3a725d78721a8ea58 P 9817b0faddf74855b5625fa520c160d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:19:19.039499 12085 raft_consensus.cc:348] T 34926e3b893f4eb3a725d78721a8ea58 P 9817b0faddf74855b5625fa520c160d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130acc74148042b68fad3073fe31193a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41629 } } peers { permanent_uuid: "f5de23594c5a401cbb3b42642de5fe2e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39947 } } peers { permanent_uuid: "9817b0faddf74855b5625fa520c160d7" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 45771 } }
I20260619 08:19:19.039655 12085 raft_consensus.cc:374] T 34926e3b893f4eb3a725d78721a8ea58 P 9817b0faddf74855b5625fa520c160d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:19.039696 12085 raft_consensus.cc:729] T 34926e3b893f4eb3a725d78721a8ea58 P 9817b0faddf74855b5625fa520c160d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9817b0faddf74855b5625fa520c160d7, State: Initialized, Role: FOLLOWER
I20260619 08:19:19.039824 12085 consensus_queue.cc:260] T 34926e3b893f4eb3a725d78721a8ea58 P 9817b0faddf74855b5625fa520c160d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130acc74148042b68fad3073fe31193a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41629 } } peers { permanent_uuid: "f5de23594c5a401cbb3b42642de5fe2e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39947 } } peers { permanent_uuid: "9817b0faddf74855b5625fa520c160d7" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 45771 } }
I20260619 08:19:19.040123 11924 heartbeater.cc:503] Master 127.4.151.190:44597 was elected leader, sending a full tablet report...
I20260619 08:19:19.040031 12084 raft_consensus.cc:348] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130acc74148042b68fad3073fe31193a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41629 } } peers { permanent_uuid: "f5de23594c5a401cbb3b42642de5fe2e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39947 } } peers { permanent_uuid: "9817b0faddf74855b5625fa520c160d7" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 45771 } }
I20260619 08:19:19.040177 12084 raft_consensus.cc:374] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:19.040210 12084 raft_consensus.cc:729] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5de23594c5a401cbb3b42642de5fe2e, State: Initialized, Role: FOLLOWER
I20260619 08:19:19.040514 12085 ts_tablet_manager.cc:1428] T 34926e3b893f4eb3a725d78721a8ea58 P 9817b0faddf74855b5625fa520c160d7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 08:19:19.040707 12088 tablet_bootstrap.cc:492] T 34926e3b893f4eb3a725d78721a8ea58 P 130acc74148042b68fad3073fe31193a: Bootstrap starting.
I20260619 08:19:19.041386 12088 tablet_bootstrap.cc:654] T 34926e3b893f4eb3a725d78721a8ea58 P 130acc74148042b68fad3073fe31193a: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:19.041414 12084 consensus_queue.cc:260] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130acc74148042b68fad3073fe31193a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41629 } } peers { permanent_uuid: "f5de23594c5a401cbb3b42642de5fe2e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39947 } } peers { permanent_uuid: "9817b0faddf74855b5625fa520c160d7" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 45771 } }
I20260619 08:19:19.041644 12084 ts_tablet_manager.cc:1428] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 08:19:19.041689 12060 heartbeater.cc:503] Master 127.4.151.190:44597 was elected leader, sending a full tablet report...
I20260619 08:19:19.041963 12088 log.cc:824] T 34926e3b893f4eb3a725d78721a8ea58 P 130acc74148042b68fad3073fe31193a: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:19.042836 12088 tablet_bootstrap.cc:492] T 34926e3b893f4eb3a725d78721a8ea58 P 130acc74148042b68fad3073fe31193a: No bootstrap required, opened a new log
I20260619 08:19:19.042919 12088 ts_tablet_manager.cc:1397] T 34926e3b893f4eb3a725d78721a8ea58 P 130acc74148042b68fad3073fe31193a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:19:19.044200 12088 raft_consensus.cc:348] T 34926e3b893f4eb3a725d78721a8ea58 P 130acc74148042b68fad3073fe31193a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130acc74148042b68fad3073fe31193a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41629 } } peers { permanent_uuid: "f5de23594c5a401cbb3b42642de5fe2e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39947 } } peers { permanent_uuid: "9817b0faddf74855b5625fa520c160d7" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 45771 } }
I20260619 08:19:19.044337 12088 raft_consensus.cc:374] T 34926e3b893f4eb3a725d78721a8ea58 P 130acc74148042b68fad3073fe31193a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:19.044382 12088 raft_consensus.cc:729] T 34926e3b893f4eb3a725d78721a8ea58 P 130acc74148042b68fad3073fe31193a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 130acc74148042b68fad3073fe31193a, State: Initialized, Role: FOLLOWER
I20260619 08:19:19.044492 12088 consensus_queue.cc:260] T 34926e3b893f4eb3a725d78721a8ea58 P 130acc74148042b68fad3073fe31193a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130acc74148042b68fad3073fe31193a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41629 } } peers { permanent_uuid: "f5de23594c5a401cbb3b42642de5fe2e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39947 } } peers { permanent_uuid: "9817b0faddf74855b5625fa520c160d7" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 45771 } }
I20260619 08:19:19.044739 12088 ts_tablet_manager.cc:1428] T 34926e3b893f4eb3a725d78721a8ea58 P 130acc74148042b68fad3073fe31193a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:19:19.044807 11787 heartbeater.cc:503] Master 127.4.151.190:44597 was elected leader, sending a full tablet report...
W20260619 08:19:19.118211 11926 tablet.cc:2369] T 34926e3b893f4eb3a725d78721a8ea58 P 9817b0faddf74855b5625fa520c160d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:19:19.135164 12090 raft_consensus.cc:482] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:19:19.135295 12090 raft_consensus.cc:504] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130acc74148042b68fad3073fe31193a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41629 } } peers { permanent_uuid: "f5de23594c5a401cbb3b42642de5fe2e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39947 } } peers { permanent_uuid: "9817b0faddf74855b5625fa520c160d7" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 45771 } }
I20260619 08:19:19.135700 12090 leader_election.cc:288] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 130acc74148042b68fad3073fe31193a (127.4.151.129:41629), 9817b0faddf74855b5625fa520c160d7 (127.4.151.130:45771)
I20260619 08:19:19.139106 12064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:19.135918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:41629 (local address 127.4.151.131:41869)
0619 08:19:19.136025 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:19.136041 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:19.136090 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:19.136342 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:19.136342 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:19.136344 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:19.136417 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:19.136421 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.137162 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.137163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:19.138004 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:19.138006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.138793 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.138795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:19.138894 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:19.138898 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:19:19.138979 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 08:19:19.139024 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":32}
I20260619 08:19:19.139104 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:19.136144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:41869 (local address 127.4.151.129:41629)
0619 08:19:19.136212 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:19.136214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:19.136223 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:19:19.136251 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:19.136252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:19.136261 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:19.136335 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:19.136529 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.137060 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.138141 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.138669 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.138753 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:19.138761 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:19.138989 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 08:19:19.139032 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 08:19:19.139317 12065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:19.135918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:45771 (local address 127.4.151.131:50911)
0619 08:19:19.135990 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:19.136003 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:19.136051 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:19.136468 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:19.136469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:19.136471 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:19.136540 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:19.136541 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.137335 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.137336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:19.138327 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:19.138330 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.139083 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.139084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:19.139158 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:19.139161 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:19:19.139194 (+ 33us) client_negotiation.cc:240] Negotiation successful
0619 08:19:19.139230 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 08:19:19.139559 12083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:19.136186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:50911 (local address 127.4.151.130:45771)
0619 08:19:19.136281 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:19.136283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:19.136293 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:19:19.136318 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:19.136319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:19.136327 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:19.136428 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:19.136625 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.137235 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.138453 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.138985 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.139418 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:19.139428 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:19.139447 (+ 19us) server_negotiation.cc:299] Negotiation successful
0619 08:19:19.139499 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 08:19:19.139720 11740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34926e3b893f4eb3a725d78721a8ea58" candidate_uuid: "f5de23594c5a401cbb3b42642de5fe2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "130acc74148042b68fad3073fe31193a" is_pre_election: true
I20260619 08:19:19.139786 11877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34926e3b893f4eb3a725d78721a8ea58" candidate_uuid: "f5de23594c5a401cbb3b42642de5fe2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9817b0faddf74855b5625fa520c160d7" is_pre_election: true
I20260619 08:19:19.139883 11740 raft_consensus.cc:2454] T 34926e3b893f4eb3a725d78721a8ea58 P 130acc74148042b68fad3073fe31193a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5de23594c5a401cbb3b42642de5fe2e in term 0.
I20260619 08:19:19.139917 11877 raft_consensus.cc:2454] T 34926e3b893f4eb3a725d78721a8ea58 P 9817b0faddf74855b5625fa520c160d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5de23594c5a401cbb3b42642de5fe2e in term 0.
I20260619 08:19:19.140128 11948 leader_election.cc:302] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 130acc74148042b68fad3073fe31193a, f5de23594c5a401cbb3b42642de5fe2e; no voters:
I20260619 08:19:19.140285 12090 raft_consensus.cc:2781] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:19:19.140357 12090 raft_consensus.cc:482] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:19:19.140389 12090 raft_consensus.cc:3037] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:19.141227 12090 raft_consensus.cc:504] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130acc74148042b68fad3073fe31193a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41629 } } peers { permanent_uuid: "f5de23594c5a401cbb3b42642de5fe2e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39947 } } peers { permanent_uuid: "9817b0faddf74855b5625fa520c160d7" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 45771 } }
I20260619 08:19:19.141547 12072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:19.138497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41502 (local address 127.4.151.188:46337)
0619 08:19:19.138555 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:19.138557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:19.138565 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:19:19.138603 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:19.138604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:19.138625 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:19.138701 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:19.138861 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.139619 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.140799 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.141201 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.141289 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:19.141365 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:19.141434 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 08:19:19.141485 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 08:19:19.141549 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:19.138405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:46337 (local address 127.0.0.1:41502)
0619 08:19:19.138455 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:19.138468 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:19.138510 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:19.138724 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:19.138725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:19.138727 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:19.138776 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:19.138778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.139716 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.139717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:19.140700 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:19.140703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.141288 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.141289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:19.141347 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:19.141350 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:19:19.141429 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 08:19:19.141479 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 08:19:19.141525 11740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34926e3b893f4eb3a725d78721a8ea58" candidate_uuid: "f5de23594c5a401cbb3b42642de5fe2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "130acc74148042b68fad3073fe31193a"
I20260619 08:19:19.141606 11740 raft_consensus.cc:3037] T 34926e3b893f4eb3a725d78721a8ea58 P 130acc74148042b68fad3073fe31193a [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:19.141620 11877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34926e3b893f4eb3a725d78721a8ea58" candidate_uuid: "f5de23594c5a401cbb3b42642de5fe2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9817b0faddf74855b5625fa520c160d7"
I20260619 08:19:19.141687 11877 raft_consensus.cc:3037] T 34926e3b893f4eb3a725d78721a8ea58 P 9817b0faddf74855b5625fa520c160d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:19.142568 11740 raft_consensus.cc:2454] T 34926e3b893f4eb3a725d78721a8ea58 P 130acc74148042b68fad3073fe31193a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5de23594c5a401cbb3b42642de5fe2e in term 1.
I20260619 08:19:19.142568 11877 raft_consensus.cc:2454] T 34926e3b893f4eb3a725d78721a8ea58 P 9817b0faddf74855b5625fa520c160d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5de23594c5a401cbb3b42642de5fe2e in term 1.
I20260619 08:19:19.142763 11948 leader_election.cc:302] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 130acc74148042b68fad3073fe31193a, f5de23594c5a401cbb3b42642de5fe2e; no voters:
I20260619 08:19:19.143009 12093 raft_consensus.cc:2781] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:19.143028 11611 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260619 08:19:19.143188 12093 raft_consensus.cc:686] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e [term 1 LEADER]: Becoming Leader. State: Replica: f5de23594c5a401cbb3b42642de5fe2e, State: Running, Role: LEADER
I20260619 08:19:19.142769 12090 leader_election.cc:288] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e [CANDIDATE]: Term 1 election: Requested vote from peers 130acc74148042b68fad3073fe31193a (127.4.151.129:41629), 9817b0faddf74855b5625fa520c160d7 (127.4.151.130:45771)
I20260619 08:19:19.143337 12093 consensus_queue.cc:237] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130acc74148042b68fad3073fe31193a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41629 } } peers { permanent_uuid: "f5de23594c5a401cbb3b42642de5fe2e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39947 } } peers { permanent_uuid: "9817b0faddf74855b5625fa520c160d7" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 45771 } }
I20260619 08:19:19.143388 11654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6dd1d19eb084efbbf3cca9f38d54286. Latest consensus state: current_term: 1 leader_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: NON_VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: true } } }
I20260619 08:19:19.143479 11654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [sys.catalog]: This master's current role is: LEARNER
I20260619 08:19:19.144589 10815 catalog_manager.cc:5495] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e reported cstate change: term changed from 0 to 1, leader changed from <none> to f5de23594c5a401cbb3b42642de5fe2e (127.4.151.131). New cstate: current_term: 1 leader_uuid: "f5de23594c5a401cbb3b42642de5fe2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130acc74148042b68fad3073fe31193a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5de23594c5a401cbb3b42642de5fe2e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9817b0faddf74855b5625fa520c160d7" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 45771 } health_report { overall_health: UNKNOWN } } }
I20260619 08:19:19.145238 11374 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: NON_VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: true } } }
I20260619 08:19:19.145331 11374 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:19.146014 11374 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286: attempting to promote NON_VOTER 4bb90fb221c1411a84e6f320351deacc to VOTER
I20260619 08:19:19.146153 11374 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: false } }
I20260619 08:19:19.146517 11611 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 1 LEARNER]: Refusing update from remote peer d6dd1d19eb084efbbf3cca9f38d54286: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260619 08:19:19.146682 11422 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 1 FOLLOWER]: Refusing update from remote peer d6dd1d19eb084efbbf3cca9f38d54286: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260619 08:19:19.146726 11571 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260619 08:19:19.146853 12101 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260619 08:19:19.147502 11571 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 4bb90fb221c1411a84e6f320351deacc (127.4.151.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: false } } }
I20260619 08:19:19.147632 11611 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 4bb90fb221c1411a84e6f320351deacc (127.4.151.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: false } } }
I20260619 08:19:19.148343 11571 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: false } } }
I20260619 08:19:19.148425 11571 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:19.148489 11571 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: false } } }
I20260619 08:19:19.148487 11654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: false } } }
I20260619 08:19:19.148552 11571 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:19.148588 11654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:19:19.148617 11571 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: false } } }
I20260619 08:19:19.148671 11571 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:19.148800 11422 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 4bb90fb221c1411a84e6f320351deacc (127.4.151.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: false } } }
I20260619 08:19:19.149735 12102 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: false } } }
I20260619 08:19:19.149853 12102 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:19:19.176275 12081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:19.173715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49028 (local address 127.4.151.131:39947)
0619 08:19:19.173785 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:19.173787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:19.173804 (+ 17us) server_negotiation.cc:407] Connection header received
0619 08:19:19.173840 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:19.173841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:19.173874 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:19.173945 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:19:19.174126 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.174683 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.175500 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.175712 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.175776 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:19.175915 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:19:19.176029 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:19:19.176101 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:19.176120 (+ 19us) server_negotiation.cc:299] Negotiation successful
0619 08:19:19.176143 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 08:19:19.178488 11877 raft_consensus.cc:1261] T 34926e3b893f4eb3a725d78721a8ea58 P 9817b0faddf74855b5625fa520c160d7 [term 1 FOLLOWER]: Refusing update from remote peer f5de23594c5a401cbb3b42642de5fe2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:19:19.178495 11740 raft_consensus.cc:1261] T 34926e3b893f4eb3a725d78721a8ea58 P 130acc74148042b68fad3073fe31193a [term 1 FOLLOWER]: Refusing update from remote peer f5de23594c5a401cbb3b42642de5fe2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:19:19.178939 12094 consensus_queue.cc:1035] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "130acc74148042b68fad3073fe31193a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:19:19.179064 12093 consensus_queue.cc:1035] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9817b0faddf74855b5625fa520c160d7" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:19:19.184360 12106 mvcc.cc:205] Tried to move back new op lower bound from 7298486923991785472 to 7298486923851939840. Current Snapshot: MvccSnapshot[applied={T|T < 7298486923991785472}]
I20260619 08:19:19.184861 10815 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:19:19.184952 10815 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:19:19.185315 12105 mvcc.cc:205] Tried to move back new op lower bound from 7298486923991785472 to 7298486923851939840. Current Snapshot: MvccSnapshot[applied={T|T < 7298486923991785472 or (T in {7298486923991785472})}]
I20260619 08:19:19.186578 10815 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6dd1d19eb084efbbf3cca9f38d54286: Sending DeleteTablet for 3 replicas of tablet 34926e3b893f4eb3a725d78721a8ea58
I20260619 08:19:19.186895 11857 tablet_service.cc:1514] Processing DeleteTablet for tablet 34926e3b893f4eb3a725d78721a8ea58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:19 UTC) from {username='oryx', principal='oryx/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:41420
I20260619 08:19:19.187045 11993 tablet_service.cc:1514] Processing DeleteTablet for tablet 34926e3b893f4eb3a725d78721a8ea58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:19 UTC) from {username='oryx', principal='oryx/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:49014
I20260619 08:19:19.187260 11720 tablet_service.cc:1514] Processing DeleteTablet for tablet 34926e3b893f4eb3a725d78721a8ea58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:19 UTC) from {username='oryx', principal='oryx/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:58726
I20260619 08:19:19.187629 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 10783
I20260619 08:19:19.187791 12115 tablet_replica.cc:330] T 34926e3b893f4eb3a725d78721a8ea58 P 130acc74148042b68fad3073fe31193a: stopping tablet replica
I20260619 08:19:19.187920 12114 tablet_replica.cc:330] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e: stopping tablet replica
I20260619 08:19:19.188094 12115 raft_consensus.cc:2229] T 34926e3b893f4eb3a725d78721a8ea58 P 130acc74148042b68fad3073fe31193a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:19:19.188226 12115 raft_consensus.cc:2258] T 34926e3b893f4eb3a725d78721a8ea58 P 130acc74148042b68fad3073fe31193a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:19.188362 12114 raft_consensus.cc:2229] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:19:19.188503 12114 raft_consensus.cc:2258] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:19.189002 12114 ts_tablet_manager.cc:1905] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:19.189265 12115 ts_tablet_manager.cc:1905] T 34926e3b893f4eb3a725d78721a8ea58 P 130acc74148042b68fad3073fe31193a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:19.190945 12113 tablet_replica.cc:330] T 34926e3b893f4eb3a725d78721a8ea58 P 9817b0faddf74855b5625fa520c160d7: stopping tablet replica
I20260619 08:19:19.191082 12113 raft_consensus.cc:2229] T 34926e3b893f4eb3a725d78721a8ea58 P 9817b0faddf74855b5625fa520c160d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:19:19.191196 12115 ts_tablet_manager.cc:1918] T 34926e3b893f4eb3a725d78721a8ea58 P 130acc74148042b68fad3073fe31193a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:19:19.191216 12113 raft_consensus.cc:2258] T 34926e3b893f4eb3a725d78721a8ea58 P 9817b0faddf74855b5625fa520c160d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:19.191320 12115 log.cc:1196] T 34926e3b893f4eb3a725d78721a8ea58 P 130acc74148042b68fad3073fe31193a: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-0/wal/wals/34926e3b893f4eb3a725d78721a8ea58
I20260619 08:19:19.191386 12114 ts_tablet_manager.cc:1918] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:19:19.191449 12114 log.cc:1196] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/34926e3b893f4eb3a725d78721a8ea58
I20260619 08:19:19.191754 12114 ts_tablet_manager.cc:1939] T 34926e3b893f4eb3a725d78721a8ea58 P f5de23594c5a401cbb3b42642de5fe2e: Deleting consensus metadata
I20260619 08:19:19.191849 12115 ts_tablet_manager.cc:1939] T 34926e3b893f4eb3a725d78721a8ea58 P 130acc74148042b68fad3073fe31193a: Deleting consensus metadata
I20260619 08:19:19.193912 12113 ts_tablet_manager.cc:1905] T 34926e3b893f4eb3a725d78721a8ea58 P 9817b0faddf74855b5625fa520c160d7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 08:19:19.195307 11949 connection.cc:535] server connection from 127.0.0.1:49014 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 08:19:19.195616 11674 connection.cc:535] server connection from 127.0.0.1:58726 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 08:19:19.195931 12113 ts_tablet_manager.cc:1918] T 34926e3b893f4eb3a725d78721a8ea58 P 9817b0faddf74855b5625fa520c160d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:19:19.196089 12113 log.cc:1196] T 34926e3b893f4eb3a725d78721a8ea58 P 9817b0faddf74855b5625fa520c160d7: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/34926e3b893f4eb3a725d78721a8ea58
I20260619 08:19:19.196473 12113 ts_tablet_manager.cc:1939] T 34926e3b893f4eb3a725d78721a8ea58 P 9817b0faddf74855b5625fa520c160d7: Deleting consensus metadata
W20260619 08:19:19.197404 11813 connection.cc:441] server connection from 127.0.0.1:41420 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41420 (request call id 1) could send its response
I20260619 08:19:19.202528 12072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:19.187467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41506 (local address 127.4.151.188:46337)
0619 08:19:19.187509 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:19.187511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:19.193761 (+ 6250us) server_negotiation.cc:407] Connection header received
0619 08:19:19.194009 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:19.194010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:19.194025 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:19.194066 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:19.194935 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.195752 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.196938 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.197256 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.197352 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:19.199125 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:19.199126 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:19.199127 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:19.199135 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:19.200899 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:19.201339 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:19.201340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:19.201341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:19.201438 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:19.201643 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:19.201644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:19.201645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:19.201782 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:19.201878 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:19.202281 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 08:19:19.202393 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:19:19.207111 11449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:19.198108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51412 (local address 127.4.151.189:44003)
0619 08:19:19.198156 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:19.198158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:19.198455 (+ 297us) server_negotiation.cc:407] Connection header received
0619 08:19:19.198568 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:19.198569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:19.198581 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:19.198611 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:19.199259 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.200340 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.201359 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.201720 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.201785 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:19.203506 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:19.203507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:19.203508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:19.203516 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:19.205555 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:19.206063 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:19.206064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:19.206065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:19.206132 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:19.206405 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:19.206406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:19.206406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:19.206558 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:19.206661 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:19.206896 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 08:19:19.206980 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 08:19:19.208513 11790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:19.198477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41508 (local address 127.4.151.188:46337)
0619 08:19:19.198521 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:19.198523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:19.199396 (+ 873us) server_negotiation.cc:407] Connection header received
0619 08:19:19.199510 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:19.199511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:19.199523 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:19.199598 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:19.201939 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.202670 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.203796 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:19.204098 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:19.204154 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:19.205614 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:19.205615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:19.205616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:19.205622 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:19.207314 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:19.207691 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:19.207692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:19.207693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:19.207757 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:19.207986 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:19.207987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:19.207987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:19.208102 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:19.208213 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:19.208318 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 08:19:19.208402 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 08:19:20.183494 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:20.183230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.129:36947)
0619 08:19:20.183341 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:20.183430 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":61,"threads_started":1}
W20260619 08:19:20.183784 11787 heartbeater.cc:644] Failed to heartbeat to 127.4.151.190:44597 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
I20260619 08:19:20.184561 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:20.184340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.131:38423)
0619 08:19:20.184452 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:20.184526 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":54,"threads_started":1}
W20260619 08:19:20.184770 12060 heartbeater.cc:644] Failed to heartbeat to 127.4.151.190:44597 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
I20260619 08:19:20.186136 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:20.185769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.130:40371)
0619 08:19:20.186019 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:20.186082 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":51,"threads_started":1}
W20260619 08:19:20.186293 11924 heartbeater.cc:644] Failed to heartbeat to 127.4.151.190:44597 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
I20260619 08:19:20.686991 12127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6dd1d19eb084efbbf3cca9f38d54286)
I20260619 08:19:20.686967 12128 raft_consensus.cc:482] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6dd1d19eb084efbbf3cca9f38d54286)
I20260619 08:19:20.687117 12127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: false } }
I20260619 08:19:20.687155 12128 raft_consensus.cc:504] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: false } }
I20260619 08:19:20.687450 12128 leader_election.cc:288] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6dd1d19eb084efbbf3cca9f38d54286 (127.4.151.190:44597), 4bb90fb221c1411a84e6f320351deacc (127.4.151.188:46337)
I20260619 08:19:20.687458 12127 leader_election.cc:288] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6dd1d19eb084efbbf3cca9f38d54286 (127.4.151.190:44597), bed09c04d0564fecbde79ff3e9cfe92f (127.4.151.189:44003)
I20260619 08:19:20.687844 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:20.687586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.0.0.1:55056)
0619 08:19:20.687738 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:20.687793 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":68,"threads_started":1}
I20260619 08:19:20.687845 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:20.687583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.0.0.1:55058)
0619 08:19:20.687745 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:20.687799 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":63,"threads_started":1}
W20260619 08:19:20.687983 11594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
W20260619 08:19:20.688040 11399 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
I20260619 08:19:20.688356 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:20.688274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.0.0.1:55072)
0619 08:19:20.688311 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:20.688332 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 08:19:20.688444 11594 leader_election.cc:334] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6dd1d19eb084efbbf3cca9f38d54286 (127.4.151.190:44597): Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
I20260619 08:19:20.688947 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:20.688831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.0.0.1:55084)
0619 08:19:20.688884 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:20.688916 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 08:19:20.689039 11399 leader_election.cc:334] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6dd1d19eb084efbbf3cca9f38d54286 (127.4.151.190:44597): Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
I20260619 08:19:20.691077 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:20.687785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:46337 (local address 127.0.0.1:37676)
0619 08:19:20.687924 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:20.687934 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:20.687985 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:20.688405 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:20.688406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:20.688408 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:20.688472 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:20.688475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:20.689425 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:20.689426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:20.690226 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:20.690228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:20.690811 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:20.690812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:20.690880 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:20.690883 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:19:20.690958 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 08:19:20.691001 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":47,"threads_started":1}
I20260619 08:19:20.691331 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:20.687646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:44003 (local address 127.0.0.1:60834)
0619 08:19:20.687833 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:20.687845 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:20.688494 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:20.688710 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:20.688711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:20.688713 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:20.688761 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:20.688763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:20.689589 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:20.689590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:20.690317 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:20.690319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:20.690932 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:20.690933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:20.690993 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:20.690998 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:19:20.691220 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 08:19:20.691270 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":38,"threads_started":1}
I20260619 08:19:20.691515 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:20.688143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37676 (local address 127.4.151.188:46337)
0619 08:19:20.688265 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:20.688267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:20.688275 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:19:20.688303 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:20.688304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:20.688316 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:20.688396 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:20.688526 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:20.689266 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:20.690325 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:20.690704 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:20.690815 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:20.690826 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:20.691306 (+ 480us) server_negotiation.cc:299] Negotiation successful
0619 08:19:20.691359 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260619 08:19:20.691182 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:20.687706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60834 (local address 127.4.151.189:44003)
0619 08:19:20.688069 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:20.688072 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:20.688082 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:19:20.688586 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:20.688587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:20.688596 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:20.688640 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:20.688847 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:20.689492 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:20.690430 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:20.690791 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:20.690884 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:20.690896 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:20.691082 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 08:19:20.691126 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":73,"threads_started":1}
I20260619 08:19:20.691802 11611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4bb90fb221c1411a84e6f320351deacc" is_pre_election: true
I20260619 08:19:20.691802 11422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4bb90fb221c1411a84e6f320351deacc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" is_pre_election: true
I20260619 08:19:20.691892 11611 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed09c04d0564fecbde79ff3e9cfe92f in term 1.
I20260619 08:19:20.691917 11422 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bb90fb221c1411a84e6f320351deacc in term 1.
I20260619 08:19:20.692044 11399 leader_election.cc:302] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4bb90fb221c1411a84e6f320351deacc, bed09c04d0564fecbde79ff3e9cfe92f; no voters: d6dd1d19eb084efbbf3cca9f38d54286
I20260619 08:19:20.692068 11593 leader_election.cc:302] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4bb90fb221c1411a84e6f320351deacc, bed09c04d0564fecbde79ff3e9cfe92f; no voters: d6dd1d19eb084efbbf3cca9f38d54286
I20260619 08:19:20.692162 12128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 08:19:20.692205 12128 raft_consensus.cc:482] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 1 FOLLOWER]: Starting leader election (detected failure of leader d6dd1d19eb084efbbf3cca9f38d54286)
I20260619 08:19:20.692207 12127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 08:19:20.692230 12128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 1 FOLLOWER]: Advancing to term 2
I20260619 08:19:20.692240 12127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 1 FOLLOWER]: Starting leader election (detected failure of leader d6dd1d19eb084efbbf3cca9f38d54286)
I20260619 08:19:20.692260 12127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 1 FOLLOWER]: Advancing to term 2
I20260619 08:19:20.693117 12127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: false } }
I20260619 08:19:20.693117 12128 raft_consensus.cc:504] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: false } }
I20260619 08:19:20.693253 12128 leader_election.cc:288] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [CANDIDATE]: Term 2 election: Requested vote from peers d6dd1d19eb084efbbf3cca9f38d54286 (127.4.151.190:44597), 4bb90fb221c1411a84e6f320351deacc (127.4.151.188:46337)
I20260619 08:19:20.693259 12127 leader_election.cc:288] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [CANDIDATE]: Term 2 election: Requested vote from peers d6dd1d19eb084efbbf3cca9f38d54286 (127.4.151.190:44597), bed09c04d0564fecbde79ff3e9cfe92f (127.4.151.189:44003)
I20260619 08:19:20.693496 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:20.693396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.0.0.1:55112)
0619 08:19:20.693437 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:20.693465 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260619 08:19:20.693549 11422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4bb90fb221c1411a84e6f320351deacc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bed09c04d0564fecbde79ff3e9cfe92f"
I20260619 08:19:20.693593 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:20.693442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.0.0.1:55128)
0619 08:19:20.693489 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:20.693516 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260619 08:19:20.693571 11611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4bb90fb221c1411a84e6f320351deacc"
I20260619 08:19:20.693667 11422 raft_consensus.cc:2379] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bb90fb221c1411a84e6f320351deacc in current term 2: Already voted for candidate bed09c04d0564fecbde79ff3e9cfe92f in this term.
I20260619 08:19:20.693711 11611 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bed09c04d0564fecbde79ff3e9cfe92f in current term 2: Already voted for candidate 4bb90fb221c1411a84e6f320351deacc in this term.
I20260619 08:19:20.693989 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:20.693889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.0.0.1:55134)
0619 08:19:20.693936 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:20.693966 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 08:19:20.694222 11399 leader_election.cc:334] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d6dd1d19eb084efbbf3cca9f38d54286 (127.4.151.190:44597): Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
I20260619 08:19:20.694262 11399 leader_election.cc:302] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bed09c04d0564fecbde79ff3e9cfe92f; no voters: 4bb90fb221c1411a84e6f320351deacc, d6dd1d19eb084efbbf3cca9f38d54286
I20260619 08:19:20.694284 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:20.694165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.0.0.1:55136)
0619 08:19:20.694218 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:20.694244 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 08:19:20.694325 12128 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260619 08:19:20.694378 11594 leader_election.cc:334] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d6dd1d19eb084efbbf3cca9f38d54286 (127.4.151.190:44597): Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
I20260619 08:19:20.694434 11594 leader_election.cc:302] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bb90fb221c1411a84e6f320351deacc; no voters: bed09c04d0564fecbde79ff3e9cfe92f, d6dd1d19eb084efbbf3cca9f38d54286
I20260619 08:19:20.694509 12127 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 08:19:21.184492 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:21.184208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.129:51939)
0619 08:19:21.184361 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:21.184428 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260619 08:19:21.185313 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:21.185139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.131:35567)
0619 08:19:21.185243 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:21.185279 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":64,"threads_started":1}
I20260619 08:19:21.186767 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:21.186564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.130:48469)
0619 08:19:21.186688 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:21.186727 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260619 08:19:22.185345 12140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.185117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.129:37495)
0619 08:19:22.185258 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.185299 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260619 08:19:22.185901 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.185714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.131:50401)
0619 08:19:22.185824 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.185859 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
I20260619 08:19:22.187361 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.187127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.130:54137)
0619 08:19:22.187250 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.187288 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":57,"threads_started":1}
I20260619 08:19:22.775130 12143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:19:22.775261 12143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: false } }
I20260619 08:19:22.775473 12143 leader_election.cc:288] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d6dd1d19eb084efbbf3cca9f38d54286 (127.4.151.190:44597), bed09c04d0564fecbde79ff3e9cfe92f (127.4.151.189:44003)
I20260619 08:19:22.775777 11422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4bb90fb221c1411a84e6f320351deacc" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" is_pre_election: true
I20260619 08:19:22.775816 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.775622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.0.0.1:55428)
0619 08:19:22.775736 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.775776 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":44,"threads_started":1}
I20260619 08:19:22.775986 11422 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bb90fb221c1411a84e6f320351deacc in term 2.
I20260619 08:19:22.776167 11593 leader_election.cc:302] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bb90fb221c1411a84e6f320351deacc, bed09c04d0564fecbde79ff3e9cfe92f; no voters:
I20260619 08:19:22.776279 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.776160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.0.0.1:55432)
0619 08:19:22.776209 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.776236 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 08:19:22.776295 12143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260619 08:19:22.776350 12143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:19:22.776371 12143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 2 FOLLOWER]: Advancing to term 3
W20260619 08:19:22.776379 11594 leader_election.cc:334] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d6dd1d19eb084efbbf3cca9f38d54286 (127.4.151.190:44597): Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
I20260619 08:19:22.777141 12143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: false } }
I20260619 08:19:22.777278 12143 leader_election.cc:288] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [CANDIDATE]: Term 3 election: Requested vote from peers d6dd1d19eb084efbbf3cca9f38d54286 (127.4.151.190:44597), bed09c04d0564fecbde79ff3e9cfe92f (127.4.151.189:44003)
I20260619 08:19:22.777477 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.777350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.0.0.1:55446)
0619 08:19:22.777405 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.777441 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 08:19:22.777463 11422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4bb90fb221c1411a84e6f320351deacc" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bed09c04d0564fecbde79ff3e9cfe92f"
I20260619 08:19:22.777534 11422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 2 FOLLOWER]: Advancing to term 3
I20260619 08:19:22.777806 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.777725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.0.0.1:55462)
0619 08:19:22.777767 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.777791 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 08:19:22.777907 11594 leader_election.cc:334] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d6dd1d19eb084efbbf3cca9f38d54286 (127.4.151.190:44597): Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
I20260619 08:19:22.778324 11422 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bb90fb221c1411a84e6f320351deacc in term 3.
I20260619 08:19:22.778498 11593 leader_election.cc:302] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4bb90fb221c1411a84e6f320351deacc, bed09c04d0564fecbde79ff3e9cfe92f; no voters: d6dd1d19eb084efbbf3cca9f38d54286
I20260619 08:19:22.778613 12143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 3 FOLLOWER]: Leader election won for term 3
I20260619 08:19:22.778770 12143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [term 3 LEADER]: Becoming Leader. State: Replica: 4bb90fb221c1411a84e6f320351deacc, State: Running, Role: LEADER
I20260619 08:19:22.778867 12143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: false } }
I20260619 08:19:22.779274 12146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bb90fb221c1411a84e6f320351deacc. Latest consensus state: current_term: 3 leader_uuid: "4bb90fb221c1411a84e6f320351deacc" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: false } } }
I20260619 08:19:22.779389 12146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:22.779608 12148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:19:22.780196 12148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:19:22.780258 12148 catalog_manager.cc:1259] Loaded cluster ID: edb1d5b47cc64477b0439e888897e14a
I20260619 08:19:22.780275 12148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:19:22.780416 12148 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:19:22.780467 12148 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc: Loaded TSK: 0
I20260619 08:19:22.780555 12148 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:19:22.790253 11608 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:19:22.791888 11608 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:19:22.793159 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.793016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.0.0.1:55468)
0619 08:19:22.793087 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.793130 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 08:19:22.793308 11422 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [term 3 FOLLOWER]: Refusing update from remote peer 4bb90fb221c1411a84e6f320351deacc: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260619 08:19:22.793629 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.793507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.0.0.1:55478)
0619 08:19:22.793553 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.793588 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 08:19:22.793637 12146 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260619 08:19:22.793835 11594 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc -> Peer d6dd1d19eb084efbbf3cca9f38d54286 (127.4.151.190:44597): Couldn't send request to peer d6dd1d19eb084efbbf3cca9f38d54286. Status: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 08:19:22.794420 12150 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bb90fb221c1411a84e6f320351deacc. Latest consensus state: current_term: 3 leader_uuid: "4bb90fb221c1411a84e6f320351deacc" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: false } } }
I20260619 08:19:22.794545 12150 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:19:22.795007 12143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "4bb90fb221c1411a84e6f320351deacc" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: false } } }
I20260619 08:19:22.795120 12143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:22.795620 12150 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "4bb90fb221c1411a84e6f320351deacc" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d6dd1d19eb084efbbf3cca9f38d54286" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44597 } } peers { permanent_uuid: "bed09c04d0564fecbde79ff3e9cfe92f" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 44003 } attrs { promote: false } } peers { permanent_uuid: "4bb90fb221c1411a84e6f320351deacc" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 46337 } attrs { promote: false } } }
I20260619 08:19:22.795755 12150 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed09c04d0564fecbde79ff3e9cfe92f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:19:22.800536 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.797085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53020 (local address 127.4.151.129:41629)
0619 08:19:22.797952 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:22.797954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:22.797961 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:19:22.797985 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:22.797986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:22.797997 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:22.798069 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:22.798211 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:22.798852 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:22.799768 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:22.800172 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:22.800255 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:22.800268 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:22.800421 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 08:19:22.800459 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":846,"thread_start_us":54,"threads_started":1}
I20260619 08:19:22.800536 12159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.796991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:41629 (local address 127.0.0.1:53020)
0619 08:19:22.797793 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.797803 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:22.797845 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:22.798087 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:22.798088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:22.798089 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:22.798138 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:22.798139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:22.798942 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:22.798943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:22.799676 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:22.799678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:22.800265 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:22.800266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:22.800328 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:22.800332 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:19:22.800414 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 08:19:22.800459 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":286,"thread_start_us":92,"threads_started":1}
I20260619 08:19:22.800722 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.796991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:45771 (local address 127.0.0.1:48746)
0619 08:19:22.797074 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.797123 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:22.797184 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:22.797399 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:22.797400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:22.797401 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:22.797464 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:22.797465 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:22.798351 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:22.798352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:22.799463 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:22.799466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:22.800490 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:22.800491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:22.800556 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:22.800559 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:19:22.800592 (+ 33us) client_negotiation.cc:240] Negotiation successful
0619 08:19:22.800640 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 08:19:22.800894 12158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.797300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44048 (local address 127.4.151.131:39947)
0619 08:19:22.797587 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:22.797589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:22.797597 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:19:22.797623 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:22.797624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:22.797634 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:22.797730 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:22.797821 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:22.798594 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:22.799830 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:22.800391 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:22.800461 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:22.800472 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:22.800753 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 08:19:22.800801 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":57,"threads_started":1}
I20260619 08:19:22.801002 12157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.797109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:39947 (local address 127.0.0.1:44048)
0619 08:19:22.797441 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.797451 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:22.797496 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:22.797692 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:22.797693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:22.797695 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:22.797744 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:22.797746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:22.798683 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:22.798683 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:22.799739 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:22.799741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:22.800500 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:22.800500 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:22.800556 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:22.800561 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:19:22.800621 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 08:19:22.800653 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":86,"threads_started":1}
I20260619 08:19:22.801074 12156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.797096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48746 (local address 127.4.151.130:45771)
0619 08:19:22.797234 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:22.797237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:22.797247 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:19:22.797274 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:22.797276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:22.797286 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:22.797364 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:22.797555 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:22.798282 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:22.799820 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:22.800391 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:22.800904 (+ 513us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:22.800920 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:22.800941 (+ 21us) server_negotiation.cc:299] Negotiation successful
0619 08:19:22.800986 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260619 08:19:22.801333 11857 tablet_service.cc:1467] Processing CreateTablet for tablet 1cfc03a5454749f69429d457f70c1640 (DEFAULT_TABLE table=test-table [id=66bd448ec08849bcb2a07a7fb4ae49e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:22.801479 11993 tablet_service.cc:1467] Processing CreateTablet for tablet 1cfc03a5454749f69429d457f70c1640 (DEFAULT_TABLE table=test-table [id=66bd448ec08849bcb2a07a7fb4ae49e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:22.801514 11857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cfc03a5454749f69429d457f70c1640. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:22.801609 11993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cfc03a5454749f69429d457f70c1640. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:22.801594 11720 tablet_service.cc:1467] Processing CreateTablet for tablet 1cfc03a5454749f69429d457f70c1640 (DEFAULT_TABLE table=test-table [id=66bd448ec08849bcb2a07a7fb4ae49e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:22.801698 11720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cfc03a5454749f69429d457f70c1640. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:22.802671 12160 tablet_bootstrap.cc:492] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7: Bootstrap starting.
I20260619 08:19:22.803073 12160 tablet_bootstrap.cc:654] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:22.803141 12162 tablet_bootstrap.cc:492] T 1cfc03a5454749f69429d457f70c1640 P 130acc74148042b68fad3073fe31193a: Bootstrap starting.
I20260619 08:19:22.803190 12161 tablet_bootstrap.cc:492] T 1cfc03a5454749f69429d457f70c1640 P f5de23594c5a401cbb3b42642de5fe2e: Bootstrap starting.
I20260619 08:19:22.803545 12161 tablet_bootstrap.cc:654] T 1cfc03a5454749f69429d457f70c1640 P f5de23594c5a401cbb3b42642de5fe2e: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:22.803557 12162 tablet_bootstrap.cc:654] T 1cfc03a5454749f69429d457f70c1640 P 130acc74148042b68fad3073fe31193a: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:22.803679 12160 tablet_bootstrap.cc:492] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7: No bootstrap required, opened a new log
I20260619 08:19:22.803731 12160 ts_tablet_manager.cc:1397] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:22.803874 12160 raft_consensus.cc:348] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130acc74148042b68fad3073fe31193a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41629 } } peers { permanent_uuid: "9817b0faddf74855b5625fa520c160d7" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 45771 } } peers { permanent_uuid: "f5de23594c5a401cbb3b42642de5fe2e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39947 } }
I20260619 08:19:22.803982 12160 raft_consensus.cc:374] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:22.804019 12160 raft_consensus.cc:729] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9817b0faddf74855b5625fa520c160d7, State: Initialized, Role: FOLLOWER
I20260619 08:19:22.804068 12160 consensus_queue.cc:260] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130acc74148042b68fad3073fe31193a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41629 } } peers { permanent_uuid: "9817b0faddf74855b5625fa520c160d7" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 45771 } } peers { permanent_uuid: "f5de23594c5a401cbb3b42642de5fe2e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39947 } }
I20260619 08:19:22.804121 12161 tablet_bootstrap.cc:492] T 1cfc03a5454749f69429d457f70c1640 P f5de23594c5a401cbb3b42642de5fe2e: No bootstrap required, opened a new log
I20260619 08:19:22.804168 12161 ts_tablet_manager.cc:1397] T 1cfc03a5454749f69429d457f70c1640 P f5de23594c5a401cbb3b42642de5fe2e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:22.804219 12162 tablet_bootstrap.cc:492] T 1cfc03a5454749f69429d457f70c1640 P 130acc74148042b68fad3073fe31193a: No bootstrap required, opened a new log
I20260619 08:19:22.804235 12160 ts_tablet_manager.cc:1428] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 08:19:22.804270 12162 ts_tablet_manager.cc:1397] T 1cfc03a5454749f69429d457f70c1640 P 130acc74148042b68fad3073fe31193a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:22.804275 12161 raft_consensus.cc:348] T 1cfc03a5454749f69429d457f70c1640 P f5de23594c5a401cbb3b42642de5fe2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130acc74148042b68fad3073fe31193a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41629 } } peers { permanent_uuid: "9817b0faddf74855b5625fa520c160d7" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 45771 } } peers { permanent_uuid: "f5de23594c5a401cbb3b42642de5fe2e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39947 } }
I20260619 08:19:22.804351 12161 raft_consensus.cc:374] T 1cfc03a5454749f69429d457f70c1640 P f5de23594c5a401cbb3b42642de5fe2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:22.804371 12161 raft_consensus.cc:729] T 1cfc03a5454749f69429d457f70c1640 P f5de23594c5a401cbb3b42642de5fe2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5de23594c5a401cbb3b42642de5fe2e, State: Initialized, Role: FOLLOWER
I20260619 08:19:22.804418 12161 consensus_queue.cc:260] T 1cfc03a5454749f69429d457f70c1640 P f5de23594c5a401cbb3b42642de5fe2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130acc74148042b68fad3073fe31193a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41629 } } peers { permanent_uuid: "9817b0faddf74855b5625fa520c160d7" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 45771 } } peers { permanent_uuid: "f5de23594c5a401cbb3b42642de5fe2e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39947 } }
I20260619 08:19:22.804418 12162 raft_consensus.cc:348] T 1cfc03a5454749f69429d457f70c1640 P 130acc74148042b68fad3073fe31193a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130acc74148042b68fad3073fe31193a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41629 } } peers { permanent_uuid: "9817b0faddf74855b5625fa520c160d7" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 45771 } } peers { permanent_uuid: "f5de23594c5a401cbb3b42642de5fe2e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39947 } }
I20260619 08:19:22.804497 12162 raft_consensus.cc:374] T 1cfc03a5454749f69429d457f70c1640 P 130acc74148042b68fad3073fe31193a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:22.804522 12162 raft_consensus.cc:729] T 1cfc03a5454749f69429d457f70c1640 P 130acc74148042b68fad3073fe31193a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 130acc74148042b68fad3073fe31193a, State: Initialized, Role: FOLLOWER
I20260619 08:19:22.804560 12161 ts_tablet_manager.cc:1428] T 1cfc03a5454749f69429d457f70c1640 P f5de23594c5a401cbb3b42642de5fe2e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:22.804564 12162 consensus_queue.cc:260] T 1cfc03a5454749f69429d457f70c1640 P 130acc74148042b68fad3073fe31193a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130acc74148042b68fad3073fe31193a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41629 } } peers { permanent_uuid: "9817b0faddf74855b5625fa520c160d7" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 45771 } } peers { permanent_uuid: "f5de23594c5a401cbb3b42642de5fe2e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39947 } }
I20260619 08:19:22.804740 12162 ts_tablet_manager.cc:1428] T 1cfc03a5454749f69429d457f70c1640 P 130acc74148042b68fad3073fe31193a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 08:19:22.804790 12168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.804549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.130:56665)
0619 08:19:22.804720 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.804760 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
I20260619 08:19:22.805044 11923 heartbeater.cc:503] Master 127.4.151.188:46337 was elected leader, sending a full tablet report...
I20260619 08:19:22.805478 12168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.805281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.130:54977)
0619 08:19:22.805423 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.805452 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260619 08:19:22.805624 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.805348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.129:54425)
0619 08:19:22.805500 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.805537 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
I20260619 08:19:22.806308 11786 heartbeater.cc:503] Master 127.4.151.188:46337 was elected leader, sending a full tablet report...
I20260619 08:19:22.806463 12171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.805682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.131:43155)
0619 08:19:22.806385 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.806424 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":67,"threads_started":1}
I20260619 08:19:22.806599 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.806409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.129:55627)
0619 08:19:22.806534 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.806574 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260619 08:19:22.891741 12166 raft_consensus.cc:482] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:19:22.891882 12166 raft_consensus.cc:504] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130acc74148042b68fad3073fe31193a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41629 } } peers { permanent_uuid: "9817b0faddf74855b5625fa520c160d7" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 45771 } } peers { permanent_uuid: "f5de23594c5a401cbb3b42642de5fe2e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39947 } }
I20260619 08:19:22.892290 12166 leader_election.cc:288] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 130acc74148042b68fad3073fe31193a (127.4.151.129:41629), f5de23594c5a401cbb3b42642de5fe2e (127.4.151.131:39947)
I20260619 08:19:22.896387 12158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.892566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:40577 (local address 127.4.151.131:39947)
0619 08:19:22.892614 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:22.892616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:22.892630 (+ 14us) server_negotiation.cc:407] Connection header received
0619 08:19:22.892731 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:22.892732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:22.892749 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:22.892800 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:22.893035 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:22.893855 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:22.895421 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:22.896007 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:22.896073 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:22.896083 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:22.896282 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 08:19:22.896320 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 08:19:22.896387 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.892551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:60555 (local address 127.4.151.129:41629)
0619 08:19:22.892598 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:22.892600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:22.892875 (+ 275us) server_negotiation.cc:407] Connection header received
0619 08:19:22.892922 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:22.892923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:22.892932 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:22.892980 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:22.893101 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:22.893855 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:22.895429 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:22.895993 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:22.896073 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:22.896083 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:22.896271 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 08:19:22.896313 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 08:19:22.896426 12168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.892511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:39947 (local address 127.4.151.130:40577)
0619 08:19:22.892574 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.892588 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:22.892655 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:22.892827 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:22.892828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:22.892830 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:22.892899 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:22.892901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:22.893968 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:22.893969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:22.895303 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:22.895306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:22.896081 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:22.896081 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:22.896175 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:22.896183 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 08:19:22.896244 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 08:19:22.896302 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":46}
I20260619 08:19:22.896437 12172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.892511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:41629 (local address 127.4.151.130:60555)
0619 08:19:22.892779 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.892806 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:22.892860 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:22.893001 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:22.893002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:22.893008 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:22.893039 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:22.893040 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:22.893968 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:22.893969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:22.895303 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:22.895306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:22.896127 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:22.896128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:22.896175 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:22.896182 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 08:19:22.896244 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 08:19:22.896302 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":147,"thread_start_us":121,"threads_started":1}
I20260619 08:19:22.896797 12013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cfc03a5454749f69429d457f70c1640" candidate_uuid: "9817b0faddf74855b5625fa520c160d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5de23594c5a401cbb3b42642de5fe2e" is_pre_election: true
I20260619 08:19:22.896807 11740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cfc03a5454749f69429d457f70c1640" candidate_uuid: "9817b0faddf74855b5625fa520c160d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "130acc74148042b68fad3073fe31193a" is_pre_election: true
I20260619 08:19:22.896914 11740 raft_consensus.cc:2454] T 1cfc03a5454749f69429d457f70c1640 P 130acc74148042b68fad3073fe31193a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9817b0faddf74855b5625fa520c160d7 in term 0.
I20260619 08:19:22.896986 12013 raft_consensus.cc:2454] T 1cfc03a5454749f69429d457f70c1640 P f5de23594c5a401cbb3b42642de5fe2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9817b0faddf74855b5625fa520c160d7 in term 0.
I20260619 08:19:22.897111 11812 leader_election.cc:302] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 130acc74148042b68fad3073fe31193a, 9817b0faddf74855b5625fa520c160d7; no voters:
I20260619 08:19:22.897235 12166 raft_consensus.cc:2781] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:19:22.897280 12166 raft_consensus.cc:482] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:19:22.897300 12166 raft_consensus.cc:3037] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:22.898054 12166 raft_consensus.cc:504] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130acc74148042b68fad3073fe31193a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41629 } } peers { permanent_uuid: "9817b0faddf74855b5625fa520c160d7" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 45771 } } peers { permanent_uuid: "f5de23594c5a401cbb3b42642de5fe2e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39947 } }
I20260619 08:19:22.898270 12166 leader_election.cc:288] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7 [CANDIDATE]: Term 1 election: Requested vote from peers 130acc74148042b68fad3073fe31193a (127.4.151.129:41629), f5de23594c5a401cbb3b42642de5fe2e (127.4.151.131:39947)
I20260619 08:19:22.898348 11740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cfc03a5454749f69429d457f70c1640" candidate_uuid: "9817b0faddf74855b5625fa520c160d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "130acc74148042b68fad3073fe31193a"
I20260619 08:19:22.898466 11740 raft_consensus.cc:3037] T 1cfc03a5454749f69429d457f70c1640 P 130acc74148042b68fad3073fe31193a [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:22.898476 12013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cfc03a5454749f69429d457f70c1640" candidate_uuid: "9817b0faddf74855b5625fa520c160d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5de23594c5a401cbb3b42642de5fe2e"
I20260619 08:19:22.898551 12013 raft_consensus.cc:3037] T 1cfc03a5454749f69429d457f70c1640 P f5de23594c5a401cbb3b42642de5fe2e [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:22.899247 11740 raft_consensus.cc:2454] T 1cfc03a5454749f69429d457f70c1640 P 130acc74148042b68fad3073fe31193a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9817b0faddf74855b5625fa520c160d7 in term 1.
I20260619 08:19:22.899333 12013 raft_consensus.cc:2454] T 1cfc03a5454749f69429d457f70c1640 P f5de23594c5a401cbb3b42642de5fe2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9817b0faddf74855b5625fa520c160d7 in term 1.
I20260619 08:19:22.899430 11812 leader_election.cc:302] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 130acc74148042b68fad3073fe31193a, 9817b0faddf74855b5625fa520c160d7; no voters:
I20260619 08:19:22.899571 12166 raft_consensus.cc:2781] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:22.899752 12166 raft_consensus.cc:686] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7 [term 1 LEADER]: Becoming Leader. State: Replica: 9817b0faddf74855b5625fa520c160d7, State: Running, Role: LEADER
I20260619 08:19:22.899940 12166 consensus_queue.cc:237] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130acc74148042b68fad3073fe31193a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41629 } } peers { permanent_uuid: "9817b0faddf74855b5625fa520c160d7" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 45771 } } peers { permanent_uuid: "f5de23594c5a401cbb3b42642de5fe2e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39947 } }
I20260619 08:19:22.900197 12172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.899986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.130:41959)
0619 08:19:22.900112 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.900164 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260619 08:19:22.900753 11609 catalog_manager.cc:5495] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9817b0faddf74855b5625fa520c160d7 (127.4.151.130). New cstate: current_term: 1 leader_uuid: "9817b0faddf74855b5625fa520c160d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130acc74148042b68fad3073fe31193a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9817b0faddf74855b5625fa520c160d7" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 45771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5de23594c5a401cbb3b42642de5fe2e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39947 } health_report { overall_health: UNKNOWN } } }
I20260619 08:19:22.918279 12156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.915857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48756 (local address 127.4.151.130:45771)
0619 08:19:22.915901 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:22.915903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:22.915914 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:19:22.916005 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:22.916006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:22.916032 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:22.916113 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:19:22.916222 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:22.916706 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:22.917514 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:22.917745 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:22.917815 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:22.917973 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:19:22.918086 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:19:22.918153 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:22.918199 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 08:19:22.918219 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:19:22.919478 11740 raft_consensus.cc:1261] T 1cfc03a5454749f69429d457f70c1640 P 130acc74148042b68fad3073fe31193a [term 1 FOLLOWER]: Refusing update from remote peer 9817b0faddf74855b5625fa520c160d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:19:22.919579 12013 raft_consensus.cc:1261] T 1cfc03a5454749f69429d457f70c1640 P f5de23594c5a401cbb3b42642de5fe2e [term 1 FOLLOWER]: Refusing update from remote peer 9817b0faddf74855b5625fa520c160d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:19:22.919755 12166 consensus_queue.cc:1035] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "130acc74148042b68fad3073fe31193a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 41629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:19:22.919881 12173 consensus_queue.cc:1035] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5de23594c5a401cbb3b42642de5fe2e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:19:22.920527 12059 heartbeater.cc:503] Master 127.4.151.188:46337 was elected leader, sending a full tablet report...
I20260619 08:19:22.920801 12171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.920684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.131:34205)
0619 08:19:22.920727 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.920767 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260619 08:19:22.920867 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.920758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.129:38673)
0619 08:19:22.920791 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.920828 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260619 08:19:22.922065 12171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.921976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.131:43881)
0619 08:19:22.922012 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.922036 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 08:19:22.922569 12172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.921985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.130:51285)
0619 08:19:22.922505 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.922537 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505}
I20260619 08:19:22.922845 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:22.922729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44597 (local address 127.4.151.129:52589)
0619 08:19:22.922790 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.922813 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 08:19:22.922916 12179 mvcc.cc:205] Tried to move back new op lower bound from 7298486939315806208 to 7298486939238887424. Current Snapshot: MvccSnapshot[applied={T|T < 7298486939315806208 or (T in {7298486939315806208})}]
I20260619 08:19:22.924414 11608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:19:22.924520 11608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:19:22.925895 11608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4bb90fb221c1411a84e6f320351deacc: Sending DeleteTablet for 3 replicas of tablet 1cfc03a5454749f69429d457f70c1640
I20260619 08:19:22.926277 11993 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cfc03a5454749f69429d457f70c1640 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:22 UTC) from {username='oryx', principal='oryx/127.4.151.188@KRBTEST.COM'} at 127.0.0.1:44048
I20260619 08:19:22.926321 11857 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cfc03a5454749f69429d457f70c1640 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:22 UTC) from {username='oryx', principal='oryx/127.4.151.188@KRBTEST.COM'} at 127.0.0.1:48746
I20260619 08:19:22.926471 12186 tablet_replica.cc:330] T 1cfc03a5454749f69429d457f70c1640 P f5de23594c5a401cbb3b42642de5fe2e: stopping tablet replica
I20260619 08:19:22.926533 12186 raft_consensus.cc:2229] T 1cfc03a5454749f69429d457f70c1640 P f5de23594c5a401cbb3b42642de5fe2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:19:22.926599 12186 raft_consensus.cc:2258] T 1cfc03a5454749f69429d457f70c1640 P f5de23594c5a401cbb3b42642de5fe2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:22.926635 11720 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cfc03a5454749f69429d457f70c1640 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:22 UTC) from {username='oryx', principal='oryx/127.4.151.188@KRBTEST.COM'} at 127.0.0.1:53020
I20260619 08:19:22.926963 12186 ts_tablet_manager.cc:1905] T 1cfc03a5454749f69429d457f70c1640 P f5de23594c5a401cbb3b42642de5fe2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:22.927167 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 11560
I20260619 08:19:22.928800 12187 tablet_replica.cc:330] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7: stopping tablet replica
I20260619 08:19:22.928866 12187 raft_consensus.cc:2229] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:19:22.928946 12187 raft_consensus.cc:2258] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:22.929219 12187 ts_tablet_manager.cc:1905] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:22.929531 12186 ts_tablet_manager.cc:1918] T 1cfc03a5454749f69429d457f70c1640 P f5de23594c5a401cbb3b42642de5fe2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:19:22.929637 12186 log.cc:1196] T 1cfc03a5454749f69429d457f70c1640 P f5de23594c5a401cbb3b42642de5fe2e: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/1cfc03a5454749f69429d457f70c1640
I20260619 08:19:22.929942 12186 ts_tablet_manager.cc:1939] T 1cfc03a5454749f69429d457f70c1640 P f5de23594c5a401cbb3b42642de5fe2e: Deleting consensus metadata
I20260619 08:19:22.930321 11592 catalog_manager.cc:4841] TS f5de23594c5a401cbb3b42642de5fe2e (127.4.151.131:39947): tablet 1cfc03a5454749f69429d457f70c1640 (table test-table [id=66bd448ec08849bcb2a07a7fb4ae49e9]) successfully deleted
I20260619 08:19:22.930922 12187 ts_tablet_manager.cc:1918] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:19:22.930989 12187 log.cc:1196] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/1cfc03a5454749f69429d457f70c1640
I20260619 08:19:22.931247 12187 ts_tablet_manager.cc:1939] T 1cfc03a5454749f69429d457f70c1640 P 9817b0faddf74855b5625fa520c160d7: Deleting consensus metadata
I20260619 08:19:22.931583 11594 catalog_manager.cc:4841] TS 9817b0faddf74855b5625fa520c160d7 (127.4.151.130:45771): tablet 1cfc03a5454749f69429d457f70c1640 (table test-table [id=66bd448ec08849bcb2a07a7fb4ae49e9]) successfully deleted
W20260619 08:19:22.933323 12157 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:22.933165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:41629 (local address 127.0.0.1:53028)
0619 08:19:22.933227 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:22.933265 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.129:41629: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 08:19:22.933494 11593 catalog_manager.cc:4584] TS 130acc74148042b68fad3073fe31193a (127.4.151.129:41629): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1cfc03a5454749f69429d457f70c1640: Network error: Client connection negotiation failed: client connection to 127.4.151.129:41629: connect: Connection reset by peer (error 104)
I20260619 08:19:22.933693 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 11794
I20260619 08:19:22.939556 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 11930
I20260619 08:19:22.945070 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 11279
I20260619 08:19:22.950186 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 11466
2026-06-19T08:19:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6785 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:19:23 dist-test-slave-d3n8 krb5kdc[12191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:19:23 dist-test-slave-d3n8 krb5kdc[12191](info): set up 2 sockets
Jun 19 08:19:23 dist-test-slave-d3n8 krb5kdc[12191](info): commencing operation
krb5kdc: starting...
W20260619 08:19:24.046236 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:19:24 dist-test-slave-d3n8 krb5kdc[12191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:19:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:19:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.190.keytab.
Entry for principal oryx/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.190.keytab.
I20260619 08:19:24.172845 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:38339
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.190.keytab
--principal=oryx/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:38339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:24.254843 12204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:24.255122 12204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:24.255162 12204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:24.256414 12204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:19:24.256496 12204 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 08:19:24.256520 12204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:24.256556 12204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:24.256611 12204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:19:24.256639 12204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:19:24.258369 12204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:38339
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.190.keytab
--principal=oryx/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:38339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:24.258694 12204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:24.258931 12204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:24.264106 12212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:24.264124 12209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:24.264156 12210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:24.264386 12204 server_base.cc:1034] running on GCE node
I20260619 08:19:24.264675 12204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:24.265003 12204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:24.266191 12204 hybrid_clock.cc:648] HybridClock initialized: now 1781857164266185 us; error 39 us; skew 500 ppm
Jun 19 08:19:24 dist-test-slave-d3n8 krb5kdc[12191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857164, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:24.268949 12204 init.cc:377] Logged in from keytab as oryx/127.4.151.190@KRBTEST.COM (short username oryx)
I20260619 08:19:24.269557 12204 webserver.cc:460] Webserver started at http://127.4.151.190:37087/ using document root <none> and password file <none>
I20260619 08:19:24.269802 12204 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:24.269869 12204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:24.270021 12204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:24.271126 12204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "32506c1f476b4b3cb615f6f2ef7a8cb8"
format_stamp: "Formatted at 2026-06-19 08:19:24 on dist-test-slave-d3n8"
server_key: "8e40a0d6351e6e46fa6a8b73a4c7189a"
server_key_iv: "5f342555f0d7d14e5eb25307d73a637e"
server_key_version: "encryptionkey@0"
I20260619 08:19:24.271571 12204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "32506c1f476b4b3cb615f6f2ef7a8cb8"
format_stamp: "Formatted at 2026-06-19 08:19:24 on dist-test-slave-d3n8"
server_key: "8e40a0d6351e6e46fa6a8b73a4c7189a"
server_key_iv: "5f342555f0d7d14e5eb25307d73a637e"
server_key_version: "encryptionkey@0"
I20260619 08:19:24.273140 12204 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:19:24.274089 12219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:24.274303 12204 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:24.274426 12204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "32506c1f476b4b3cb615f6f2ef7a8cb8"
format_stamp: "Formatted at 2026-06-19 08:19:24 on dist-test-slave-d3n8"
server_key: "8e40a0d6351e6e46fa6a8b73a4c7189a"
server_key_iv: "5f342555f0d7d14e5eb25307d73a637e"
server_key_version: "encryptionkey@0"
I20260619 08:19:24.274508 12204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:24.293704 12204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:24.296175 12204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:24.296355 12204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:24.301234 12204 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:38339
I20260619 08:19:24.301365 12281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:38339 every 8 connection(s)
I20260619 08:19:24.301745 12204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:19:24.302471 12282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:24.304832 12282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8: Bootstrap starting.
I20260619 08:19:24.305472 12282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:24.305730 12282 log.cc:824] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:24.306479 12282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8: No bootstrap required, opened a new log
I20260619 08:19:24.308056 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 12204
I20260619 08:19:24.308179 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:19:24.308382 4702 external_mini_cluster.cc:1428] Setting key a46a8afc1f34446cd040a1598eed32b0
I20260619 08:19:24.308651 12282 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32506c1f476b4b3cb615f6f2ef7a8cb8" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38339 } }
I20260619 08:19:24.308846 12282 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:24.308888 12282 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32506c1f476b4b3cb615f6f2ef7a8cb8, State: Initialized, Role: FOLLOWER
I20260619 08:19:24.309044 12282 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32506c1f476b4b3cb615f6f2ef7a8cb8" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38339 } }
I20260619 08:19:24.309171 12282 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:19:24.309253 12282 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:19:24.309336 12282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:24.310284 12282 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32506c1f476b4b3cb615f6f2ef7a8cb8" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38339 } }
I20260619 08:19:24.310405 12282 leader_election.cc:302] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32506c1f476b4b3cb615f6f2ef7a8cb8; no voters:
I20260619 08:19:24.310621 12282 leader_election.cc:288] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:19:24.310627 12287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:24.310799 12287 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8 [term 1 LEADER]: Becoming Leader. State: Replica: 32506c1f476b4b3cb615f6f2ef7a8cb8, State: Running, Role: LEADER
I20260619 08:19:24.310881 12287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32506c1f476b4b3cb615f6f2ef7a8cb8" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38339 } }
I20260619 08:19:24.310949 12282 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:19:24.311255 12287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32506c1f476b4b3cb615f6f2ef7a8cb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32506c1f476b4b3cb615f6f2ef7a8cb8" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38339 } } }
I20260619 08:19:24.311419 12287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:24.311623 12287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32506c1f476b4b3cb615f6f2ef7a8cb8. Latest consensus state: current_term: 1 leader_uuid: "32506c1f476b4b3cb615f6f2ef7a8cb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32506c1f476b4b3cb615f6f2ef7a8cb8" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38339 } } }
I20260619 08:19:24.311699 12287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:24.311987 12294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:19:24.312558 12294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:19:24.314162 12294 catalog_manager.cc:1347] Generated new cluster ID: a1b307a81e1d44639951339fa22e20f7
I20260619 08:19:24.314208 12294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:19:24 dist-test-slave-d3n8 krb5kdc[12191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM
I20260619 08:19:24.321848 12286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:24.308971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52948 (local address 127.4.151.190:38339)
0619 08:19:24.309149 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:24.309162 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:24.309203 (+ 41us) server_negotiation.cc:407] Connection header received
0619 08:19:24.309328 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:24.309334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:24.309430 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:24.309561 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:24.310449 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.312004 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.313330 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.314010 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.314153 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:24.316978 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:24.316983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:24.316983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:24.316995 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:24.319619 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:24.320049 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:24.320052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:24.320053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:24.320185 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:24.320405 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:24.320406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:24.320406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:24.320737 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:24.321365 (+ 628us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:24.321488 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 08:19:24.321631 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
I20260619 08:19:24.353848 12294 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:19:24.354450 12294 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:19:24.362157 12294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8: Generated new TSK 0
I20260619 08:19:24.362361 12294 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.151.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.151.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.151.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.129.keytab.
Entry for principal oryx/127.4.151.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.129.keytab.
I20260619 08:19:24.401551 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:38339
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.129.keytab
--principal=oryx/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:24.503458 12308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:24.503700 12308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:24.503737 12308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:24.504928 12308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:24.504995 12308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:24.505033 12308 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 08:19:24.505071 12308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:19:24.506619 12308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.129.keytab
--principal=oryx/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.151.190:38339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:24.506894 12308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:24.507150 12308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:24.512434 12314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:24.512429 12313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:24.512638 12308 server_base.cc:1034] running on GCE node
W20260619 08:19:24.512456 12316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:24.513021 12308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:24.513330 12308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:24.514540 12308 hybrid_clock.cc:648] HybridClock initialized: now 1781857164514523 us; error 32 us; skew 500 ppm
Jun 19 08:19:24 dist-test-slave-d3n8 krb5kdc[12191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857164, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:24.517488 12308 init.cc:377] Logged in from keytab as oryx/127.4.151.129@KRBTEST.COM (short username oryx)
I20260619 08:19:24.517925 12308 webserver.cc:460] Webserver started at http://127.4.151.129:33051/ using document root <none> and password file <none>
I20260619 08:19:24.518144 12308 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:24.518206 12308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:24.518312 12308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:24.519452 12308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "3e2b5b8712524982b806e6f1686dcb2a"
format_stamp: "Formatted at 2026-06-19 08:19:24 on dist-test-slave-d3n8"
server_key: "140d76f283e85a744fe6e72c2c8f76db"
server_key_iv: "4690002cbbd45ade08bde7a35e946bee"
server_key_version: "encryptionkey@0"
I20260619 08:19:24.519790 12308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "3e2b5b8712524982b806e6f1686dcb2a"
format_stamp: "Formatted at 2026-06-19 08:19:24 on dist-test-slave-d3n8"
server_key: "140d76f283e85a744fe6e72c2c8f76db"
server_key_iv: "4690002cbbd45ade08bde7a35e946bee"
server_key_version: "encryptionkey@0"
I20260619 08:19:24.521206 12308 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:19:24.522266 12323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:24.522516 12308 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:24.522611 12308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "3e2b5b8712524982b806e6f1686dcb2a"
format_stamp: "Formatted at 2026-06-19 08:19:24 on dist-test-slave-d3n8"
server_key: "140d76f283e85a744fe6e72c2c8f76db"
server_key_iv: "4690002cbbd45ade08bde7a35e946bee"
server_key_version: "encryptionkey@0"
I20260619 08:19:24.522697 12308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:24.560122 12308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:24.562659 12308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:24.562780 12308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:24.563257 12308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:24.563298 12308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:24.563323 12308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:24.563335 12308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:24.570334 12308 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:44231
I20260619 08:19:24.570416 12438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:44231 every 8 connection(s)
Jun 19 08:19:24 dist-test-slave-d3n8 krb5kdc[12191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857164, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM
I20260619 08:19:24.570765 12308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
I20260619 08:19:24.574178 12286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:24.563985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:33789 (local address 127.4.151.190:38339)
0619 08:19:24.564036 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:24.564040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:24.564303 (+ 263us) server_negotiation.cc:407] Connection header received
0619 08:19:24.564979 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:24.564989 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:24.565004 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:24.565094 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:24.565839 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.566680 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.568323 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.568558 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.568633 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:24.571182 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:24.571184 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:24.571184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:24.571194 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:24.572721 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:24.573144 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:24.573145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:24.573146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:24.573211 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:24.573516 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:24.573517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:24.573518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:24.573639 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:24.573717 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:24.574003 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 08:19:24.574089 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 08:19:24.574360 12332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:24.563942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:38339 (local address 127.4.151.129:33789)
0619 08:19:24.564139 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:24.564175 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:24.564839 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:24.565176 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:24.565182 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:24.565545 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:24.565733 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:24.565737 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.566812 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.566814 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:24.568222 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:24.568225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.568656 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.568657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:24.568823 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:24.569463 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:24.569473 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:24.571063 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:24.572828 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:24.572836 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:24.572838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:24.573067 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:24.573278 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:24.573279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:24.573279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:24.573444 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:24.573707 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:24.573714 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:24.573851 (+ 137us) client_negotiation.cc:769] Sending connection context
0619 08:19:24.573968 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 08:19:24.574122 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":97,"threads_started":1}
I20260619 08:19:24.574970 12439 heartbeater.cc:347] Connected to a master server at 127.4.151.190:38339
I20260619 08:19:24.575048 12439 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:24.575232 12439 heartbeater.cc:511] Master 127.4.151.190:38339 requested a full tablet report, sending...
I20260619 08:19:24.575836 12235 ts_manager.cc:194] Registered new tserver with Master: 3e2b5b8712524982b806e6f1686dcb2a (127.4.151.129:44231)
I20260619 08:19:24.576484 12235 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.151.129@KRBTEST.COM'} at 127.4.151.129:33789
I20260619 08:19:24.577862 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 12308
I20260619 08:19:24.577979 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:19:24.578195 4702 external_mini_cluster.cc:1428] Setting key 3e275cd8a9c2705e65cccd0606a55cf1
I20260619 08:19:24.583209 12286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:24.575952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:52375 (local address 127.4.151.190:38339)
0619 08:19:24.575990 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:24.575992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:24.576756 (+ 764us) server_negotiation.cc:407] Connection header received
0619 08:19:24.576824 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:24.576826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:24.576836 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:24.576887 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:24.577218 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.577893 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.578832 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.579107 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.579166 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:24.580367 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:24.580369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:24.580369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:24.580377 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:24.582085 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:24.582507 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:24.582508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:24.582508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:24.582556 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:24.582762 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:24.582763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:24.582763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:24.582854 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:24.582903 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:24.583084 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 08:19:24.583146 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:19:24.583331 12445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:24.575886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:38339 (local address 127.4.151.129:52375)
0619 08:19:24.576703 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:24.576714 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:24.576771 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:24.576918 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:24.576919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:24.577084 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:24.577137 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:24.577138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.578015 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.578016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:24.578744 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:24.578747 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.579200 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.579201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:24.579272 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:24.579826 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:24.579832 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:24.580260 (+ 428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:24.582178 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:24.582179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:24.582180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:24.582417 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:24.582623 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:24.582624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:24.582625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:24.582700 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:24.582924 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:24.582925 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:24.582978 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 08:19:24.583056 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 08:19:24.583175 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for oryx/127.4.151.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.151.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.151.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.130.keytab.
Entry for principal oryx/127.4.151.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.130.keytab.
I20260619 08:19:24.606381 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:38339
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.130.keytab
--principal=oryx/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:24.689126 12448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:24.689388 12448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:24.689461 12448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:24.690654 12448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:24.690730 12448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:24.690766 12448 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 08:19:24.690807 12448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:19:24.692394 12448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.130.keytab
--principal=oryx/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.151.190:38339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:24.692674 12448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:24.692921 12448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:24.698205 12454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:24.698257 12456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:24.698225 12453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:24.698469 12448 server_base.cc:1034] running on GCE node
I20260619 08:19:24.698828 12448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:24.699124 12448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:24.700313 12448 hybrid_clock.cc:648] HybridClock initialized: now 1781857164700313 us; error 45 us; skew 500 ppm
Jun 19 08:19:24 dist-test-slave-d3n8 krb5kdc[12191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857164, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:24.703240 12448 init.cc:377] Logged in from keytab as oryx/127.4.151.130@KRBTEST.COM (short username oryx)
I20260619 08:19:24.703766 12448 webserver.cc:460] Webserver started at http://127.4.151.130:42801/ using document root <none> and password file <none>
I20260619 08:19:24.704028 12448 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:24.704105 12448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:24.704258 12448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:24.705343 12448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "0174ead94fee4075b70c3cde1ea725be"
format_stamp: "Formatted at 2026-06-19 08:19:24 on dist-test-slave-d3n8"
server_key: "888c4b9fd960d3381ba2a68c5fcd0b00"
server_key_iv: "975ac8d2391592812e36231be46205a6"
server_key_version: "encryptionkey@0"
I20260619 08:19:24.705782 12448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "0174ead94fee4075b70c3cde1ea725be"
format_stamp: "Formatted at 2026-06-19 08:19:24 on dist-test-slave-d3n8"
server_key: "888c4b9fd960d3381ba2a68c5fcd0b00"
server_key_iv: "975ac8d2391592812e36231be46205a6"
server_key_version: "encryptionkey@0"
I20260619 08:19:24.707183 12448 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:24.708065 12463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:24.708309 12448 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:24.708438 12448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "0174ead94fee4075b70c3cde1ea725be"
format_stamp: "Formatted at 2026-06-19 08:19:24 on dist-test-slave-d3n8"
server_key: "888c4b9fd960d3381ba2a68c5fcd0b00"
server_key_iv: "975ac8d2391592812e36231be46205a6"
server_key_version: "encryptionkey@0"
I20260619 08:19:24.708544 12448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:24.728223 12448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:24.730715 12448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:24.730901 12448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:24.731554 12448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:24.731606 12448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:24.731643 12448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:24.731669 12448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 08:19:24 dist-test-slave-d3n8 krb5kdc[12191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857164, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM
I20260619 08:19:24.739295 12448 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:36549
I20260619 08:19:24.739382 12578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:36549 every 8 connection(s)
I20260619 08:19:24.739750 12448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
I20260619 08:19:24.741580 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 12448
I20260619 08:19:24.741705 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:19:24.741918 12286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:24.732120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:38541 (local address 127.4.151.190:38339)
0619 08:19:24.732183 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:24.732185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:24.732340 (+ 155us) server_negotiation.cc:407] Connection header received
0619 08:19:24.733133 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:24.733135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:24.733147 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:24.733248 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:24.734268 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.734813 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.735963 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.736180 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.736265 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:24.738637 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:24.738639 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:24.738639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:24.738648 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:24.740389 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:24.740805 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:24.740806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:24.740807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:24.740869 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:24.741113 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:24.741114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:24.741114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:24.741228 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:24.741283 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:24.741752 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 08:19:24.741844 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 08:19:24.741945 4702 external_mini_cluster.cc:1428] Setting key a2a661b5f34af91231888ca675e7212a
I20260619 08:19:24.742105 12472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:24.731987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:38339 (local address 127.4.151.130:38541)
0619 08:19:24.732192 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:24.732240 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:24.732976 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:24.733357 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:24.733362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:24.733794 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:24.734139 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:24.734145 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.735022 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.735023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:24.735847 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:24.735849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.736288 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.736289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:24.736430 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:24.737021 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:24.737032 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:24.738524 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:24.740471 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:24.740476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:24.740478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:24.740700 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:24.740928 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:24.740928 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:24.740928 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:24.741020 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:24.741298 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:24.741302 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:24.741524 (+ 222us) client_negotiation.cc:769] Sending connection context
0619 08:19:24.741658 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 08:19:24.741863 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":103,"threads_started":1}
I20260619 08:19:24.742590 12579 heartbeater.cc:347] Connected to a master server at 127.4.151.190:38339
I20260619 08:19:24.742674 12579 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:24.742870 12579 heartbeater.cc:511] Master 127.4.151.190:38339 requested a full tablet report, sending...
I20260619 08:19:24.743287 12235 ts_manager.cc:194] Registered new tserver with Master: 0174ead94fee4075b70c3cde1ea725be (127.4.151.130:36549)
I20260619 08:19:24.743830 12235 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.151.130@KRBTEST.COM'} at 127.4.151.130:38541
I20260619 08:19:24.750738 12585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:24.743673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:38339 (local address 127.4.151.130:51861)
0619 08:19:24.743830 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:24.743841 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:24.743899 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:24.744148 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:24.744148 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:24.744302 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:24.744362 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:24.744364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.745398 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.745399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:24.746196 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:24.746199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.746708 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.746709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:24.746759 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:24.747241 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:24.747246 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:24.747633 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:24.749606 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:24.749607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:24.749607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:24.749772 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:24.750040 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:24.750040 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:24.750041 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:24.750112 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:24.750473 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:24.750474 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:24.750510 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 08:19:24.750585 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 08:19:24.750665 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260619 08:19:24.750810 12286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:24.743963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:51861 (local address 127.4.151.190:38339)
0619 08:19:24.744007 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:24.744009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:24.744019 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:19:24.744040 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:24.744040 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:24.744059 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:24.744133 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:24.744437 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.745273 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.746287 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.746621 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.746704 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:24.747740 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:24.747742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:24.747742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:24.747750 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:24.749524 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:24.749859 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:24.749860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:24.749861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:24.749936 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:24.750185 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:24.750186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:24.750186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:24.750398 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:24.750480 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:24.750623 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 08:19:24.750698 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
WARNING: no policy specified for oryx/127.4.151.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.151.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.151.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.131.keytab.
Entry for principal oryx/127.4.151.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.131.keytab.
I20260619 08:19:24.773442 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:38339
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.131.keytab
--principal=oryx/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:24.858008 12588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:24.858225 12588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:24.858249 12588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:24.859436 12588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:24.859479 12588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:24.859499 12588 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 08:19:24.859517 12588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:19:24.861140 12588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.131.keytab
--principal=oryx/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.151.190:38339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:24.861423 12588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:24.861673 12588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:24.867643 12594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:24.867787 12588 server_base.cc:1034] running on GCE node
W20260619 08:19:24.867643 12593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:24.867712 12596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:24.868290 12588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:24.868520 12588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:24.869683 12588 hybrid_clock.cc:648] HybridClock initialized: now 1781857164869663 us; error 44 us; skew 500 ppm
Jun 19 08:19:24 dist-test-slave-d3n8 krb5kdc[12191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857164, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:24.872563 12588 init.cc:377] Logged in from keytab as oryx/127.4.151.131@KRBTEST.COM (short username oryx)
I20260619 08:19:24.873003 12588 webserver.cc:460] Webserver started at http://127.4.151.131:45099/ using document root <none> and password file <none>
I20260619 08:19:24.873690 12588 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:24.873756 12588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:24.873978 12588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:24.875500 12588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "5f5e8cd5233e492f86f8e30c5f287be6"
format_stamp: "Formatted at 2026-06-19 08:19:24 on dist-test-slave-d3n8"
server_key: "fb34b6a5df3ac1ced2b2a852b5a38c57"
server_key_iv: "a39a28da8d7145f13979f133531bf190"
server_key_version: "encryptionkey@0"
I20260619 08:19:24.875977 12588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "5f5e8cd5233e492f86f8e30c5f287be6"
format_stamp: "Formatted at 2026-06-19 08:19:24 on dist-test-slave-d3n8"
server_key: "fb34b6a5df3ac1ced2b2a852b5a38c57"
server_key_iv: "a39a28da8d7145f13979f133531bf190"
server_key_version: "encryptionkey@0"
I20260619 08:19:24.877739 12588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260619 08:19:24.878701 12603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:24.878947 12588 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:24.879053 12588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "5f5e8cd5233e492f86f8e30c5f287be6"
format_stamp: "Formatted at 2026-06-19 08:19:24 on dist-test-slave-d3n8"
server_key: "fb34b6a5df3ac1ced2b2a852b5a38c57"
server_key_iv: "a39a28da8d7145f13979f133531bf190"
server_key_version: "encryptionkey@0"
I20260619 08:19:24.879117 12588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:24.898727 12588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:24.901124 12588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:24.901245 12588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:24.901857 12588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:24.901902 12588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:24.901929 12588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:24.901942 12588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:24.908823 12588 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:41127
I20260619 08:19:24.909091 12718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:41127 every 8 connection(s)
I20260619 08:19:24.909310 12588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
Jun 19 08:19:24 dist-test-slave-d3n8 krb5kdc[12191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857164, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM
I20260619 08:19:24.913616 12286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:24.902511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:34177 (local address 127.4.151.190:38339)
0619 08:19:24.902579 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:24.902581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:24.902776 (+ 195us) server_negotiation.cc:407] Connection header received
0619 08:19:24.903451 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:24.903453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:24.903469 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:24.903558 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:24.904444 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.905311 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.906918 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.907256 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.907347 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:24.910447 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:24.910448 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:24.910448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:24.910456 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:24.912068 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:24.912493 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:24.912495 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:24.912495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:24.912578 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:24.912872 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:24.912873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:24.912874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:24.913024 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:24.913096 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:24.913433 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 08:19:24.913547 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 08:19:24.913777 12612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:24.902487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:38339 (local address 127.4.151.131:34177)
0619 08:19:24.902654 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:24.902690 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:24.903297 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:24.903644 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:24.903648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:24.904042 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:24.904333 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:24.904339 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.905458 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.905459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:24.906829 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:24.906832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.907364 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.907365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:24.907533 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:24.908293 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:24.908313 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:24.910342 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:24.912156 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:24.912164 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:24.912166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:24.912403 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:24.912659 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:24.912660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:24.912660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:24.912779 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:24.913110 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:24.913116 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:24.913246 (+ 130us) client_negotiation.cc:769] Sending connection context
0619 08:19:24.913347 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 08:19:24.913552 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260619 08:19:24.914353 12719 heartbeater.cc:347] Connected to a master server at 127.4.151.190:38339
I20260619 08:19:24.914480 12719 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:24.914680 12719 heartbeater.cc:511] Master 127.4.151.190:38339 requested a full tablet report, sending...
I20260619 08:19:24.915417 12234 ts_manager.cc:194] Registered new tserver with Master: 5f5e8cd5233e492f86f8e30c5f287be6 (127.4.151.131:41127)
I20260619 08:19:24.915849 12234 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.151.131@KRBTEST.COM'} at 127.4.151.131:34177
I20260619 08:19:24.918998 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 12588
I20260619 08:19:24.919111 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:19:24.919328 4702 external_mini_cluster.cc:1428] Setting key d11e9c8ff510ebe4f89882789f89a67d
I20260619 08:19:24.920742 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:19:24.922569 12725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:24.915461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:38339 (local address 127.4.151.131:38385)
0619 08:19:24.916027 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:24.916040 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:24.916100 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:24.916226 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:24.916227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:24.916439 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:24.916505 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:24.916507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.917223 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.917224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:24.917798 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:24.917800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.918156 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.918156 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:24.918206 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:24.918728 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:24.918733 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:24.919012 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:24.921354 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:24.921355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:24.921355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:24.921607 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:24.921886 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:24.921886 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:24.921887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:24.921951 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:24.922289 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:24.922289 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:24.922337 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 08:19:24.922400 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 08:19:24.922493 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":65,"threads_started":1}
I20260619 08:19:24.922672 12286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:24.915551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:38385 (local address 127.4.151.190:38339)
0619 08:19:24.915596 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:24.915598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:24.916111 (+ 513us) server_negotiation.cc:407] Connection header received
0619 08:19:24.916137 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:24.916138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:24.916148 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:24.916208 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:24.916589 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.917097 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.917895 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.918080 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.918148 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:24.919145 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:24.919146 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:24.919147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:24.919154 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:24.921265 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:24.921685 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:24.921686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:24.921686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:24.921753 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:24.922062 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:24.922063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:24.922064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:24.922218 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:24.922289 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:24.922448 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 08:19:24.922534 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Jun 19 08:19:24 dist-test-slave-d3n8 krb5kdc[12191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857164, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 08:19:24 dist-test-slave-d3n8 krb5kdc[12191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:24 dist-test-slave-d3n8 krb5kdc[12191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
W20260619 08:19:24.943747 12286 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:24.937873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52950 (local address 127.4.151.190:38339)
0619 08:19:24.938018 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:24.938019 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:24.938026 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:19:24.938050 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:24.938051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:24.938059 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:24.938132 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:24.938780 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.939276 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.940027 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.940224 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.940317 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:24.943644 (+ 3327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52950: BlockingRecv error: recv got EOF from 127.0.0.1:52950 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
Jun 19 08:19:24 dist-test-slave-d3n8 krb5kdc[12191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857164, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM
I20260619 08:19:24.951854 12286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:24.944523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52964 (local address 127.4.151.190:38339)
0619 08:19:24.944577 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:24.944579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:24.944629 (+ 50us) server_negotiation.cc:407] Connection header received
0619 08:19:24.944682 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:24.944683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:24.944691 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:24.944742 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:24.945228 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.945838 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.946553 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.946752 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.946816 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:24.949162 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:24.949164 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:24.949164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:24.949171 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:24.950691 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:24.951055 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:24.951056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:24.951057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:24.951121 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:24.951344 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:24.951345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:24.951345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:24.951461 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:24.951541 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:24.951690 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 08:19:24.951752 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 08:19:24.953840 12234 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:19:24.955469 12234 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:19:24.962291 12741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:24.957963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:44231 (local address 127.0.0.1:45296)
0619 08:19:24.958210 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:24.958252 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:24.958332 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:24.958527 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:24.958529 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:24.958531 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:24.958726 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:24.958729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.960112 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.960113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:24.960992 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:24.960995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.961978 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.961979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:24.962064 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:24.962077 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 08:19:24.962150 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 08:19:24.962202 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":53,"thread_start_us":87,"threads_started":1}
I20260619 08:19:24.962350 12743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:24.958047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45296 (local address 127.4.151.129:44231)
0619 08:19:24.958186 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:24.958191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:24.958312 (+ 121us) server_negotiation.cc:407] Connection header received
0619 08:19:24.958414 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:24.958416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:24.958431 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:24.958504 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:24.958793 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.959950 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.961091 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.961800 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.961868 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:24.961942 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:24.962175 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 08:19:24.962229 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":68,"threads_started":1}
I20260619 08:19:24.963085 12373 tablet_service.cc:1467] Processing CreateTablet for tablet 76530e8155ac42a3a711efe437496d5e (DEFAULT_TABLE table=test-table [id=21bb03646619435f881f936955bbffb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:24.963527 12373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76530e8155ac42a3a711efe437496d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:24.963600 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:24.958386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49872 (local address 127.4.151.131:41127)
0619 08:19:24.958641 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:24.958645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:24.959052 (+ 407us) server_negotiation.cc:407] Connection header received
0619 08:19:24.959141 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:24.959143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:24.959159 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:24.959263 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:24.959358 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.960164 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.961659 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.962415 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.962511 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:24.962590 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:24.963457 (+ 867us) server_negotiation.cc:299] Negotiation successful
0619 08:19:24.963510 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":114,"threads_started":1}
I20260619 08:19:24.963591 12745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:24.958178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:41127 (local address 127.0.0.1:49872)
0619 08:19:24.958974 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:24.958984 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:24.959057 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:24.959256 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:24.959257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:24.959259 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:24.959298 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:24.959299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.960266 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.960267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:24.961563 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:24.961566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.963310 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.963311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:24.963357 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:24.963361 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:19:24.963440 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 08:19:24.963487 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":65,"threads_started":1}
I20260619 08:19:24.963598 12742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:24.957966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:36549 (local address 127.0.0.1:43624)
0619 08:19:24.958257 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:24.958269 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:24.958334 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:24.958672 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:24.958673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:24.958675 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:24.958800 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:24.958803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.960043 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.960044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:24.961564 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:24.961566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.963262 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.963264 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:24.963339 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:24.963344 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:19:24.963415 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 08:19:24.963470 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":46,"thread_start_us":62,"threads_started":1}
I20260619 08:19:24.963847 12744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:24.958072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43624 (local address 127.4.151.130:36549)
0619 08:19:24.958275 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:24.958279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:24.958424 (+ 145us) server_negotiation.cc:407] Connection header received
0619 08:19:24.958475 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:24.958479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:24.958497 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:24.958572 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:24.958876 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.959951 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.962629 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:24.963150 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:24.963230 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:24.963277 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:24.963738 (+ 461us) server_negotiation.cc:299] Negotiation successful
0619 08:19:24.963774 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":90,"threads_started":1}
I20260619 08:19:24.965893 12653 tablet_service.cc:1467] Processing CreateTablet for tablet 76530e8155ac42a3a711efe437496d5e (DEFAULT_TABLE table=test-table [id=21bb03646619435f881f936955bbffb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:24.966007 12513 tablet_service.cc:1467] Processing CreateTablet for tablet 76530e8155ac42a3a711efe437496d5e (DEFAULT_TABLE table=test-table [id=21bb03646619435f881f936955bbffb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:24.966300 12653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76530e8155ac42a3a711efe437496d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:24.966367 12513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76530e8155ac42a3a711efe437496d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:24.968464 12747 tablet_bootstrap.cc:492] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a: Bootstrap starting.
I20260619 08:19:24.969211 12747 tablet_bootstrap.cc:654] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:24.969539 12748 tablet_bootstrap.cc:492] T 76530e8155ac42a3a711efe437496d5e P 5f5e8cd5233e492f86f8e30c5f287be6: Bootstrap starting.
I20260619 08:19:24.969595 12747 log.cc:824] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:24.970207 12748 tablet_bootstrap.cc:654] T 76530e8155ac42a3a711efe437496d5e P 5f5e8cd5233e492f86f8e30c5f287be6: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:24.970402 12747 tablet_bootstrap.cc:492] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a: No bootstrap required, opened a new log
I20260619 08:19:24.970490 12747 ts_tablet_manager.cc:1397] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 08:19:24.970597 12748 log.cc:824] T 76530e8155ac42a3a711efe437496d5e P 5f5e8cd5233e492f86f8e30c5f287be6: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:24.971194 12750 tablet_bootstrap.cc:492] T 76530e8155ac42a3a711efe437496d5e P 0174ead94fee4075b70c3cde1ea725be: Bootstrap starting.
I20260619 08:19:24.971336 12748 tablet_bootstrap.cc:492] T 76530e8155ac42a3a711efe437496d5e P 5f5e8cd5233e492f86f8e30c5f287be6: No bootstrap required, opened a new log
I20260619 08:19:24.971421 12748 ts_tablet_manager.cc:1397] T 76530e8155ac42a3a711efe437496d5e P 5f5e8cd5233e492f86f8e30c5f287be6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:19:24.971918 12750 tablet_bootstrap.cc:654] T 76530e8155ac42a3a711efe437496d5e P 0174ead94fee4075b70c3cde1ea725be: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:24.972272 12747 raft_consensus.cc:348] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } } peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } }
I20260619 08:19:24.972393 12747 raft_consensus.cc:374] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:24.972414 12747 raft_consensus.cc:729] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2b5b8712524982b806e6f1686dcb2a, State: Initialized, Role: FOLLOWER
I20260619 08:19:24.972533 12747 consensus_queue.cc:260] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } } peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } }
I20260619 08:19:24.972792 12747 ts_tablet_manager.cc:1428] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 08:19:24.974196 12748 raft_consensus.cc:348] T 76530e8155ac42a3a711efe437496d5e P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } } peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } }
I20260619 08:19:24.974414 12748 raft_consensus.cc:374] T 76530e8155ac42a3a711efe437496d5e P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:24.974463 12748 raft_consensus.cc:729] T 76530e8155ac42a3a711efe437496d5e P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f5e8cd5233e492f86f8e30c5f287be6, State: Initialized, Role: FOLLOWER
I20260619 08:19:24.974735 12439 heartbeater.cc:503] Master 127.4.151.190:38339 was elected leader, sending a full tablet report...
I20260619 08:19:24.974730 12748 consensus_queue.cc:260] T 76530e8155ac42a3a711efe437496d5e P 5f5e8cd5233e492f86f8e30c5f287be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } } peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } }
I20260619 08:19:24.975059 12719 heartbeater.cc:503] Master 127.4.151.190:38339 was elected leader, sending a full tablet report...
I20260619 08:19:24.975057 12748 ts_tablet_manager.cc:1428] T 76530e8155ac42a3a711efe437496d5e P 5f5e8cd5233e492f86f8e30c5f287be6: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 08:19:24.979715 12750 log.cc:824] T 76530e8155ac42a3a711efe437496d5e P 0174ead94fee4075b70c3cde1ea725be: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:24.980749 12750 tablet_bootstrap.cc:492] T 76530e8155ac42a3a711efe437496d5e P 0174ead94fee4075b70c3cde1ea725be: No bootstrap required, opened a new log
I20260619 08:19:24.980842 12750 ts_tablet_manager.cc:1397] T 76530e8155ac42a3a711efe437496d5e P 0174ead94fee4075b70c3cde1ea725be: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260619 08:19:24.982450 12750 raft_consensus.cc:348] T 76530e8155ac42a3a711efe437496d5e P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } } peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } }
I20260619 08:19:24.982625 12750 raft_consensus.cc:374] T 76530e8155ac42a3a711efe437496d5e P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:24.982668 12750 raft_consensus.cc:729] T 76530e8155ac42a3a711efe437496d5e P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0174ead94fee4075b70c3cde1ea725be, State: Initialized, Role: FOLLOWER
I20260619 08:19:24.982798 12750 consensus_queue.cc:260] T 76530e8155ac42a3a711efe437496d5e P 0174ead94fee4075b70c3cde1ea725be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } } peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } }
I20260619 08:19:24.983085 12750 ts_tablet_manager.cc:1428] T 76530e8155ac42a3a711efe437496d5e P 0174ead94fee4075b70c3cde1ea725be: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 08:19:24.983163 12579 heartbeater.cc:503] Master 127.4.151.190:38339 was elected leader, sending a full tablet report...
W20260619 08:19:24.990410 12580 tablet.cc:2369] T 76530e8155ac42a3a711efe437496d5e P 0174ead94fee4075b70c3cde1ea725be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:19:25.071609 12440 tablet.cc:2369] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:19:25.160054 12720 tablet.cc:2369] T 76530e8155ac42a3a711efe437496d5e P 5f5e8cd5233e492f86f8e30c5f287be6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:19:25.400693 12752 raft_consensus.cc:482] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:19:25.400835 12752 raft_consensus.cc:504] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } } peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } }
I20260619 08:19:25.401163 12752 leader_election.cc:288] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0174ead94fee4075b70c3cde1ea725be (127.4.151.130:36549), 5f5e8cd5233e492f86f8e30c5f287be6 (127.4.151.131:41127)
I20260619 08:19:25.404471 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:25.401708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:57143 (local address 127.4.151.131:41127)
0619 08:19:25.401766 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:25.401768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:25.401776 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:19:25.401872 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:25.401873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:25.401882 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:25.402057 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:25.402220 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.402723 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.403734 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.404101 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.404156 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:25.404164 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:25.404347 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 08:19:25.404396 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 08:19:25.404465 12757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:25.401606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:41127 (local address 127.4.151.129:57143)
0619 08:19:25.401708 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:25.401718 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:25.401786 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:25.402089 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:25.402090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:25.402093 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:25.402143 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:25.402144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.402828 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.402829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:25.403622 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:25.403624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.404187 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.404188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:25.404260 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:25.404264 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:19:25.404318 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 08:19:25.404350 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":1,"thread_start_us":46,"threads_started":1}
I20260619 08:19:25.404553 12744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:25.401487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:56239 (local address 127.4.151.130:36549)
0619 08:19:25.401550 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:25.401552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:25.401627 (+ 75us) server_negotiation.cc:407] Connection header received
0619 08:19:25.401766 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:25.401768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:25.401779 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:25.401837 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:25.402039 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.402602 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.403739 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.404122 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.404219 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:25.404228 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:25.404432 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 08:19:25.404477 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 08:19:25.404596 12756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:25.401361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:36549 (local address 127.4.151.129:56239)
0619 08:19:25.401545 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:25.401556 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:25.401621 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:25.401856 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:25.401864 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:25.401867 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:25.401961 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:25.401963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.402688 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.402689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:25.403620 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:25.403623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.404238 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.404239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:25.404339 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:25.404343 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:19:25.404402 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 08:19:25.404459 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":35,"thread_start_us":82,"threads_started":1}
I20260619 08:19:25.405076 12673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76530e8155ac42a3a711efe437496d5e" candidate_uuid: "3e2b5b8712524982b806e6f1686dcb2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" is_pre_election: true
I20260619 08:19:25.405081 12533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76530e8155ac42a3a711efe437496d5e" candidate_uuid: "3e2b5b8712524982b806e6f1686dcb2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0174ead94fee4075b70c3cde1ea725be" is_pre_election: true
I20260619 08:19:25.405234 12673 raft_consensus.cc:2454] T 76530e8155ac42a3a711efe437496d5e P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e2b5b8712524982b806e6f1686dcb2a in term 0.
I20260619 08:19:25.405238 12533 raft_consensus.cc:2454] T 76530e8155ac42a3a711efe437496d5e P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e2b5b8712524982b806e6f1686dcb2a in term 0.
I20260619 08:19:25.405493 12327 leader_election.cc:302] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0174ead94fee4075b70c3cde1ea725be, 3e2b5b8712524982b806e6f1686dcb2a; no voters:
I20260619 08:19:25.405664 12752 raft_consensus.cc:2781] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:19:25.405738 12752 raft_consensus.cc:482] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:19:25.405774 12752 raft_consensus.cc:3037] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:25.406467 12752 raft_consensus.cc:504] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } } peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } }
I20260619 08:19:25.406617 12752 leader_election.cc:288] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a [CANDIDATE]: Term 1 election: Requested vote from peers 0174ead94fee4075b70c3cde1ea725be (127.4.151.130:36549), 5f5e8cd5233e492f86f8e30c5f287be6 (127.4.151.131:41127)
I20260619 08:19:25.406801 12533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76530e8155ac42a3a711efe437496d5e" candidate_uuid: "3e2b5b8712524982b806e6f1686dcb2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0174ead94fee4075b70c3cde1ea725be"
I20260619 08:19:25.406895 12533 raft_consensus.cc:3037] T 76530e8155ac42a3a711efe437496d5e P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:25.406844 12673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76530e8155ac42a3a711efe437496d5e" candidate_uuid: "3e2b5b8712524982b806e6f1686dcb2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f5e8cd5233e492f86f8e30c5f287be6"
I20260619 08:19:25.407073 12673 raft_consensus.cc:3037] T 76530e8155ac42a3a711efe437496d5e P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:25.407603 12533 raft_consensus.cc:2454] T 76530e8155ac42a3a711efe437496d5e P 0174ead94fee4075b70c3cde1ea725be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e2b5b8712524982b806e6f1686dcb2a in term 1.
I20260619 08:19:25.407804 12327 leader_election.cc:302] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0174ead94fee4075b70c3cde1ea725be, 3e2b5b8712524982b806e6f1686dcb2a; no voters:
I20260619 08:19:25.407878 12673 raft_consensus.cc:2454] T 76530e8155ac42a3a711efe437496d5e P 5f5e8cd5233e492f86f8e30c5f287be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e2b5b8712524982b806e6f1686dcb2a in term 1.
I20260619 08:19:25.407920 12752 raft_consensus.cc:2781] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:25.408052 12752 raft_consensus.cc:686] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a [term 1 LEADER]: Becoming Leader. State: Replica: 3e2b5b8712524982b806e6f1686dcb2a, State: Running, Role: LEADER
I20260619 08:19:25.408155 12752 consensus_queue.cc:237] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } } peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } }
I20260619 08:19:25.409094 12234 catalog_manager.cc:5495] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e2b5b8712524982b806e6f1686dcb2a (127.4.151.129). New cstate: current_term: 1 leader_uuid: "3e2b5b8712524982b806e6f1686dcb2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } health_report { overall_health: HEALTHY } } }
I20260619 08:19:25.515165 12762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:25.512350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45306 (local address 127.4.151.129:44231)
0619 08:19:25.512470 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:25.512473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:25.512482 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:19:25.512602 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:25.512603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:25.512627 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:25.512731 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:19:25.512884 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.513432 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.514303 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.514548 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.514632 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:25.514803 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:19:25.514925 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:19:25.515022 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:25.515041 (+ 19us) server_negotiation.cc:299] Negotiation successful
0619 08:19:25.515064 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
I20260619 08:19:25.517036 12533 raft_consensus.cc:1261] T 76530e8155ac42a3a711efe437496d5e P 0174ead94fee4075b70c3cde1ea725be [term 1 FOLLOWER]: Refusing update from remote peer 3e2b5b8712524982b806e6f1686dcb2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:19:25.517030 12673 raft_consensus.cc:1261] T 76530e8155ac42a3a711efe437496d5e P 5f5e8cd5233e492f86f8e30c5f287be6 [term 1 FOLLOWER]: Refusing update from remote peer 3e2b5b8712524982b806e6f1686dcb2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:19:25.517457 12752 consensus_queue.cc:1035] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:19:25.517716 12758 consensus_queue.cc:1035] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:19:25.519884 12763 mvcc.cc:205] Tried to move back new op lower bound from 7298486949954932736 to 7298486949512224768. Current Snapshot: MvccSnapshot[applied={T|T < 7298486949954932736}]
I20260619 08:19:25.520090 12766 mvcc.cc:205] Tried to move back new op lower bound from 7298486949954932736 to 7298486949512224768. Current Snapshot: MvccSnapshot[applied={T|T < 7298486949954932736}]
I20260619 08:19:25.520246 12768 mvcc.cc:205] Tried to move back new op lower bound from 7298486949954932736 to 7298486949512224768. Current Snapshot: MvccSnapshot[applied={T|T < 7298486949954932736}]
I20260619 08:19:25.522070 12234 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:19:25.522151 12234 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:19:25.522931 12234 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8: Sending DeleteTablet for 3 replicas of tablet 76530e8155ac42a3a711efe437496d5e
I20260619 08:19:25.523341 12513 tablet_service.cc:1514] Processing DeleteTablet for tablet 76530e8155ac42a3a711efe437496d5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:25 UTC) from {username='oryx', principal='oryx/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:43624
I20260619 08:19:25.523375 12653 tablet_service.cc:1514] Processing DeleteTablet for tablet 76530e8155ac42a3a711efe437496d5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:25 UTC) from {username='oryx', principal='oryx/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:49872
I20260619 08:19:25.523527 12234 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 08:19:25.523636 12373 tablet_service.cc:1514] Processing DeleteTablet for tablet 76530e8155ac42a3a711efe437496d5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:25 UTC) from {username='oryx', principal='oryx/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:45296
W20260619 08:19:25.523677 12234 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:19:25.523707 12776 tablet_replica.cc:330] T 76530e8155ac42a3a711efe437496d5e P 0174ead94fee4075b70c3cde1ea725be: stopping tablet replica
I20260619 08:19:25.523795 12776 raft_consensus.cc:2229] T 76530e8155ac42a3a711efe437496d5e P 0174ead94fee4075b70c3cde1ea725be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:19:25.523923 12776 raft_consensus.cc:2258] T 76530e8155ac42a3a711efe437496d5e P 0174ead94fee4075b70c3cde1ea725be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:25.524330 12776 ts_tablet_manager.cc:1905] T 76530e8155ac42a3a711efe437496d5e P 0174ead94fee4075b70c3cde1ea725be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:25.525127 12373 tablet_service.cc:1467] Processing CreateTablet for tablet f1ae6647d3774578825a68b1b04ccf87 (DEFAULT_TABLE table=test-table [id=775cab1044ca4c668c890ad325b8e4d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:25.525156 12513 tablet_service.cc:1467] Processing CreateTablet for tablet f1ae6647d3774578825a68b1b04ccf87 (DEFAULT_TABLE table=test-table [id=775cab1044ca4c668c890ad325b8e4d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:25.525266 12513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1ae6647d3774578825a68b1b04ccf87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:25.525287 12373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1ae6647d3774578825a68b1b04ccf87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:25.526340 12777 tablet_replica.cc:330] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a: stopping tablet replica
I20260619 08:19:25.526434 12777 raft_consensus.cc:2229] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:19:25.526530 12776 ts_tablet_manager.cc:1918] T 76530e8155ac42a3a711efe437496d5e P 0174ead94fee4075b70c3cde1ea725be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:19:25.526572 12777 raft_consensus.cc:2258] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:25.526599 12776 log.cc:1196] T 76530e8155ac42a3a711efe437496d5e P 0174ead94fee4075b70c3cde1ea725be: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/76530e8155ac42a3a711efe437496d5e
I20260619 08:19:25.526721 12653 tablet_service.cc:1467] Processing CreateTablet for tablet f1ae6647d3774578825a68b1b04ccf87 (DEFAULT_TABLE table=test-table [id=775cab1044ca4c668c890ad325b8e4d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:25.526830 12653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1ae6647d3774578825a68b1b04ccf87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:25.526885 12776 ts_tablet_manager.cc:1939] T 76530e8155ac42a3a711efe437496d5e P 0174ead94fee4075b70c3cde1ea725be: Deleting consensus metadata
I20260619 08:19:25.527235 12223 catalog_manager.cc:4841] TS 0174ead94fee4075b70c3cde1ea725be (127.4.151.130:36549): tablet 76530e8155ac42a3a711efe437496d5e (table test-table [id=21bb03646619435f881f936955bbffb2]) successfully deleted
I20260619 08:19:25.527151 12778 tablet_bootstrap.cc:492] T f1ae6647d3774578825a68b1b04ccf87 P 0174ead94fee4075b70c3cde1ea725be: Bootstrap starting.
I20260619 08:19:25.527686 12778 tablet_bootstrap.cc:654] T f1ae6647d3774578825a68b1b04ccf87 P 0174ead94fee4075b70c3cde1ea725be: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:25.528044 12775 tablet_replica.cc:330] T 76530e8155ac42a3a711efe437496d5e P 5f5e8cd5233e492f86f8e30c5f287be6: stopping tablet replica
I20260619 08:19:25.528141 12775 raft_consensus.cc:2229] T 76530e8155ac42a3a711efe437496d5e P 5f5e8cd5233e492f86f8e30c5f287be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:19:25.528242 12775 raft_consensus.cc:2258] T 76530e8155ac42a3a711efe437496d5e P 5f5e8cd5233e492f86f8e30c5f287be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:25.528311 12781 tablet_bootstrap.cc:492] T f1ae6647d3774578825a68b1b04ccf87 P 5f5e8cd5233e492f86f8e30c5f287be6: Bootstrap starting.
I20260619 08:19:25.528698 12781 tablet_bootstrap.cc:654] T f1ae6647d3774578825a68b1b04ccf87 P 5f5e8cd5233e492f86f8e30c5f287be6: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:25.528708 12777 ts_tablet_manager.cc:1905] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:25.528800 12778 tablet_bootstrap.cc:492] T f1ae6647d3774578825a68b1b04ccf87 P 0174ead94fee4075b70c3cde1ea725be: No bootstrap required, opened a new log
I20260619 08:19:25.528879 12778 ts_tablet_manager.cc:1397] T f1ae6647d3774578825a68b1b04ccf87 P 0174ead94fee4075b70c3cde1ea725be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:19:25.529067 12778 raft_consensus.cc:348] T f1ae6647d3774578825a68b1b04ccf87 P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.529201 12778 raft_consensus.cc:374] T f1ae6647d3774578825a68b1b04ccf87 P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:25.529260 12778 raft_consensus.cc:729] T f1ae6647d3774578825a68b1b04ccf87 P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0174ead94fee4075b70c3cde1ea725be, State: Initialized, Role: FOLLOWER
I20260619 08:19:25.529366 12778 consensus_queue.cc:260] T f1ae6647d3774578825a68b1b04ccf87 P 0174ead94fee4075b70c3cde1ea725be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.529515 12779 tablet_bootstrap.cc:492] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a: Bootstrap starting.
I20260619 08:19:25.529605 12778 ts_tablet_manager.cc:1428] T f1ae6647d3774578825a68b1b04ccf87 P 0174ead94fee4075b70c3cde1ea725be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 08:19:25.529973 12779 tablet_bootstrap.cc:654] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:25.530022 12775 ts_tablet_manager.cc:1905] T 76530e8155ac42a3a711efe437496d5e P 5f5e8cd5233e492f86f8e30c5f287be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:25.530354 12777 ts_tablet_manager.cc:1918] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:19:25.530433 12777 log.cc:1196] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-0/wal/wals/76530e8155ac42a3a711efe437496d5e
I20260619 08:19:25.530701 12777 ts_tablet_manager.cc:1939] T 76530e8155ac42a3a711efe437496d5e P 3e2b5b8712524982b806e6f1686dcb2a: Deleting consensus metadata
I20260619 08:19:25.531195 12781 tablet_bootstrap.cc:492] T f1ae6647d3774578825a68b1b04ccf87 P 5f5e8cd5233e492f86f8e30c5f287be6: No bootstrap required, opened a new log
I20260619 08:19:25.531286 12781 ts_tablet_manager.cc:1397] T f1ae6647d3774578825a68b1b04ccf87 P 5f5e8cd5233e492f86f8e30c5f287be6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 08:19:25.531352 12222 catalog_manager.cc:4841] TS 3e2b5b8712524982b806e6f1686dcb2a (127.4.151.129:44231): tablet 76530e8155ac42a3a711efe437496d5e (table test-table [id=21bb03646619435f881f936955bbffb2]) successfully deleted
I20260619 08:19:25.531675 12781 raft_consensus.cc:348] T f1ae6647d3774578825a68b1b04ccf87 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.531764 12781 raft_consensus.cc:374] T f1ae6647d3774578825a68b1b04ccf87 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:25.531786 12781 raft_consensus.cc:729] T f1ae6647d3774578825a68b1b04ccf87 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f5e8cd5233e492f86f8e30c5f287be6, State: Initialized, Role: FOLLOWER
I20260619 08:19:25.531836 12781 consensus_queue.cc:260] T f1ae6647d3774578825a68b1b04ccf87 P 5f5e8cd5233e492f86f8e30c5f287be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.531942 12775 ts_tablet_manager.cc:1918] T 76530e8155ac42a3a711efe437496d5e P 5f5e8cd5233e492f86f8e30c5f287be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:19:25.531978 12781 ts_tablet_manager.cc:1428] T f1ae6647d3774578825a68b1b04ccf87 P 5f5e8cd5233e492f86f8e30c5f287be6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 08:19:25.532016 12775 log.cc:1196] T 76530e8155ac42a3a711efe437496d5e P 5f5e8cd5233e492f86f8e30c5f287be6: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/76530e8155ac42a3a711efe437496d5e
I20260619 08:19:25.532269 12775 ts_tablet_manager.cc:1939] T 76530e8155ac42a3a711efe437496d5e P 5f5e8cd5233e492f86f8e30c5f287be6: Deleting consensus metadata
I20260619 08:19:25.532670 12223 catalog_manager.cc:4841] TS 5f5e8cd5233e492f86f8e30c5f287be6 (127.4.151.131:41127): tablet 76530e8155ac42a3a711efe437496d5e (table test-table [id=21bb03646619435f881f936955bbffb2]) successfully deleted
I20260619 08:19:25.533018 12779 tablet_bootstrap.cc:492] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a: No bootstrap required, opened a new log
I20260619 08:19:25.533095 12779 ts_tablet_manager.cc:1397] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 08:19:25.533246 12779 raft_consensus.cc:348] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.533320 12779 raft_consensus.cc:374] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:25.533353 12779 raft_consensus.cc:729] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2b5b8712524982b806e6f1686dcb2a, State: Initialized, Role: FOLLOWER
I20260619 08:19:25.533439 12779 consensus_queue.cc:260] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.533564 12779 ts_tablet_manager.cc:1428] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 08:19:25.570482 12771 raft_consensus.cc:482] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:19:25.570595 12771 raft_consensus.cc:504] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.570766 12771 leader_election.cc:288] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0174ead94fee4075b70c3cde1ea725be (127.4.151.130:36549), 5f5e8cd5233e492f86f8e30c5f287be6 (127.4.151.131:41127)
I20260619 08:19:25.570990 12533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ae6647d3774578825a68b1b04ccf87" candidate_uuid: "3e2b5b8712524982b806e6f1686dcb2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0174ead94fee4075b70c3cde1ea725be" is_pre_election: true
I20260619 08:19:25.571089 12533 raft_consensus.cc:2454] T f1ae6647d3774578825a68b1b04ccf87 P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e2b5b8712524982b806e6f1686dcb2a in term 0.
I20260619 08:19:25.571089 12673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ae6647d3774578825a68b1b04ccf87" candidate_uuid: "3e2b5b8712524982b806e6f1686dcb2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" is_pre_election: true
I20260619 08:19:25.571179 12673 raft_consensus.cc:2454] T f1ae6647d3774578825a68b1b04ccf87 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e2b5b8712524982b806e6f1686dcb2a in term 0.
I20260619 08:19:25.571230 12327 leader_election.cc:302] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0174ead94fee4075b70c3cde1ea725be, 3e2b5b8712524982b806e6f1686dcb2a; no voters:
I20260619 08:19:25.571388 12771 raft_consensus.cc:2781] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:19:25.571416 12771 raft_consensus.cc:482] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:19:25.571446 12771 raft_consensus.cc:3037] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:25.572305 12771 raft_consensus.cc:504] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.572460 12771 leader_election.cc:288] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a [CANDIDATE]: Term 1 election: Requested vote from peers 0174ead94fee4075b70c3cde1ea725be (127.4.151.130:36549), 5f5e8cd5233e492f86f8e30c5f287be6 (127.4.151.131:41127)
I20260619 08:19:25.572675 12533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ae6647d3774578825a68b1b04ccf87" candidate_uuid: "3e2b5b8712524982b806e6f1686dcb2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0174ead94fee4075b70c3cde1ea725be"
I20260619 08:19:25.572691 12673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ae6647d3774578825a68b1b04ccf87" candidate_uuid: "3e2b5b8712524982b806e6f1686dcb2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f5e8cd5233e492f86f8e30c5f287be6"
I20260619 08:19:25.572752 12533 raft_consensus.cc:3037] T f1ae6647d3774578825a68b1b04ccf87 P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:25.572757 12673 raft_consensus.cc:3037] T f1ae6647d3774578825a68b1b04ccf87 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:25.573547 12673 raft_consensus.cc:2454] T f1ae6647d3774578825a68b1b04ccf87 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e2b5b8712524982b806e6f1686dcb2a in term 1.
I20260619 08:19:25.573701 12327 leader_election.cc:302] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e2b5b8712524982b806e6f1686dcb2a, 5f5e8cd5233e492f86f8e30c5f287be6; no voters:
I20260619 08:19:25.573774 12771 raft_consensus.cc:2781] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:25.573812 12771 raft_consensus.cc:686] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a [term 1 LEADER]: Becoming Leader. State: Replica: 3e2b5b8712524982b806e6f1686dcb2a, State: Running, Role: LEADER
I20260619 08:19:25.573851 12771 consensus_queue.cc:237] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.573936 12533 raft_consensus.cc:2454] T f1ae6647d3774578825a68b1b04ccf87 P 0174ead94fee4075b70c3cde1ea725be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e2b5b8712524982b806e6f1686dcb2a in term 1.
I20260619 08:19:25.574458 12235 catalog_manager.cc:5495] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e2b5b8712524982b806e6f1686dcb2a (127.4.151.129). New cstate: current_term: 1 leader_uuid: "3e2b5b8712524982b806e6f1686dcb2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } health_report { overall_health: UNKNOWN } } }
Jun 19 08:19:25 dist-test-slave-d3n8 krb5kdc[12191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857164, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.190@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM
I20260619 08:19:25.610565 12791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:25.602387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52970 (local address 127.4.151.190:38339)
0619 08:19:25.602549 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:25.602551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:25.602563 (+ 12us) server_negotiation.cc:407] Connection header received
0619 08:19:25.602622 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:25.602623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:25.602635 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:25.602702 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:25.603196 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.603788 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.604552 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.604801 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.604883 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:25.607289 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:25.607290 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:25.607291 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:25.607299 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:25.609204 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:25.609702 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:25.609703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:25.609704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:25.609766 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:25.609967 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:25.609967 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:25.609968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:25.610083 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:25.610163 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:25.610315 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 08:19:25.610423 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":103,"threads_started":1}
I20260619 08:19:25.610841 12790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:25.602299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:38339 (local address 127.0.0.1:52970)
0619 08:19:25.602466 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:25.602480 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:25.602553 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:25.602720 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:25.602721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:25.603050 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:25.603097 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:25.603099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.603874 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.603875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:25.604459 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:25.604461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.604881 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.604882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:25.604954 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:25.605650 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:25.605655 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:25.607188 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:25.609354 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:25.609363 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:25.609365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:25.609609 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:25.609834 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:25.609835 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:25.609835 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:25.609898 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:25.610151 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:25.610157 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:25.610202 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 08:19:25.610283 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 08:19:25.610549 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":37,"thread_start_us":103,"threads_started":1}
I20260619 08:19:25.613238 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:52970:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 08:19:25.613551 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:19:25.615288 12373 tablet_service.cc:1467] Processing CreateTablet for tablet 8e1556be58ab41148e5f129b2d5a4d82 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9a4041b8bc97447aad0239aca4735eaa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 08:19:25.615368 12653 tablet_service.cc:1467] Processing CreateTablet for tablet 8e1556be58ab41148e5f129b2d5a4d82 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9a4041b8bc97447aad0239aca4735eaa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 08:19:25.615473 12373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e1556be58ab41148e5f129b2d5a4d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:25.615545 12653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e1556be58ab41148e5f129b2d5a4d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:25.616653 12779 tablet_bootstrap.cc:492] T 8e1556be58ab41148e5f129b2d5a4d82 P 3e2b5b8712524982b806e6f1686dcb2a: Bootstrap starting.
I20260619 08:19:25.617026 12779 tablet_bootstrap.cc:654] T 8e1556be58ab41148e5f129b2d5a4d82 P 3e2b5b8712524982b806e6f1686dcb2a: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:25.617110 12781 tablet_bootstrap.cc:492] T 8e1556be58ab41148e5f129b2d5a4d82 P 5f5e8cd5233e492f86f8e30c5f287be6: Bootstrap starting.
I20260619 08:19:25.617561 12781 tablet_bootstrap.cc:654] T 8e1556be58ab41148e5f129b2d5a4d82 P 5f5e8cd5233e492f86f8e30c5f287be6: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:25.617695 12779 tablet_bootstrap.cc:492] T 8e1556be58ab41148e5f129b2d5a4d82 P 3e2b5b8712524982b806e6f1686dcb2a: No bootstrap required, opened a new log
I20260619 08:19:25.617748 12779 ts_tablet_manager.cc:1397] T 8e1556be58ab41148e5f129b2d5a4d82 P 3e2b5b8712524982b806e6f1686dcb2a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:25.617931 12779 raft_consensus.cc:348] T 8e1556be58ab41148e5f129b2d5a4d82 P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.618026 12779 raft_consensus.cc:374] T 8e1556be58ab41148e5f129b2d5a4d82 P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:25.618057 12779 raft_consensus.cc:729] T 8e1556be58ab41148e5f129b2d5a4d82 P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2b5b8712524982b806e6f1686dcb2a, State: Initialized, Role: FOLLOWER
I20260619 08:19:25.618104 12779 consensus_queue.cc:260] T 8e1556be58ab41148e5f129b2d5a4d82 P 3e2b5b8712524982b806e6f1686dcb2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.618093 12513 tablet_service.cc:1467] Processing CreateTablet for tablet 8e1556be58ab41148e5f129b2d5a4d82 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9a4041b8bc97447aad0239aca4735eaa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 08:19:25.618225 12513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e1556be58ab41148e5f129b2d5a4d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:25.618229 12779 ts_tablet_manager.cc:1428] T 8e1556be58ab41148e5f129b2d5a4d82 P 3e2b5b8712524982b806e6f1686dcb2a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:25.619792 12758 tablet_replica.cc:439] T 8e1556be58ab41148e5f129b2d5a4d82 P 3e2b5b8712524982b806e6f1686dcb2a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } } }
I20260619 08:19:25.619903 12758 tablet_replica.cc:442] T 8e1556be58ab41148e5f129b2d5a4d82 P 3e2b5b8712524982b806e6f1686dcb2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:19:25.620069 12778 tablet_bootstrap.cc:492] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be: Bootstrap starting.
I20260619 08:19:25.620524 12778 tablet_bootstrap.cc:654] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:25.621172 12781 tablet_bootstrap.cc:492] T 8e1556be58ab41148e5f129b2d5a4d82 P 5f5e8cd5233e492f86f8e30c5f287be6: No bootstrap required, opened a new log
I20260619 08:19:25.621229 12781 ts_tablet_manager.cc:1397] T 8e1556be58ab41148e5f129b2d5a4d82 P 5f5e8cd5233e492f86f8e30c5f287be6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260619 08:19:25.621402 12781 raft_consensus.cc:348] T 8e1556be58ab41148e5f129b2d5a4d82 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.621491 12781 raft_consensus.cc:374] T 8e1556be58ab41148e5f129b2d5a4d82 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:25.621513 12781 raft_consensus.cc:729] T 8e1556be58ab41148e5f129b2d5a4d82 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f5e8cd5233e492f86f8e30c5f287be6, State: Initialized, Role: FOLLOWER
I20260619 08:19:25.621573 12781 consensus_queue.cc:260] T 8e1556be58ab41148e5f129b2d5a4d82 P 5f5e8cd5233e492f86f8e30c5f287be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.621587 12778 tablet_bootstrap.cc:492] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be: No bootstrap required, opened a new log
I20260619 08:19:25.621649 12778 ts_tablet_manager.cc:1397] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:19:25.621665 12781 ts_tablet_manager.cc:1428] T 8e1556be58ab41148e5f129b2d5a4d82 P 5f5e8cd5233e492f86f8e30c5f287be6: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 08:19:25.621816 12764 tablet_replica.cc:439] T 8e1556be58ab41148e5f129b2d5a4d82 P 5f5e8cd5233e492f86f8e30c5f287be6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } } }
I20260619 08:19:25.621793 12778 raft_consensus.cc:348] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.621883 12764 tablet_replica.cc:442] T 8e1556be58ab41148e5f129b2d5a4d82 P 5f5e8cd5233e492f86f8e30c5f287be6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:19:25.621910 12778 raft_consensus.cc:374] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:25.621937 12778 raft_consensus.cc:729] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0174ead94fee4075b70c3cde1ea725be, State: Initialized, Role: FOLLOWER
I20260619 08:19:25.622056 12778 consensus_queue.cc:260] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.622193 12778 ts_tablet_manager.cc:1428] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 08:19:25.622237 12765 tablet_replica.cc:439] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } } }
I20260619 08:19:25.622334 12765 tablet_replica.cc:442] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:19:25.676375 12765 raft_consensus.cc:482] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:19:25.676498 12765 raft_consensus.cc:504] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.676843 12765 leader_election.cc:288] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e2b5b8712524982b806e6f1686dcb2a (127.4.151.129:44231), 5f5e8cd5233e492f86f8e30c5f287be6 (127.4.151.131:41127)
I20260619 08:19:25.680563 12796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:25.677030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:44231 (local address 127.4.151.130:54207)
0619 08:19:25.677307 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:25.677318 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:25.677482 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:25.677704 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:25.677705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:25.677707 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:25.677802 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:25.677804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.678559 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.678560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:25.679507 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:25.679510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.680252 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.680253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:25.680341 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:25.680345 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:19:25.680403 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 08:19:25.680436 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":23,"thread_start_us":127,"threads_started":1}
I20260619 08:19:25.680562 12762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:25.677172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:54207 (local address 127.4.151.129:44231)
0619 08:19:25.677235 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:25.677237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:25.677447 (+ 210us) server_negotiation.cc:407] Connection header received
0619 08:19:25.677586 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:25.677587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:25.677600 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:25.677681 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:25.677892 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.678468 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.679637 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.680151 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.680219 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:25.680229 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:25.680432 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 08:19:25.680488 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 08:19:25.680627 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:25.677181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:45151 (local address 127.4.151.131:41127)
0619 08:19:25.677237 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:25.677239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:25.677491 (+ 252us) server_negotiation.cc:407] Connection header received
0619 08:19:25.677587 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:25.677588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:25.677601 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:25.677678 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:25.677922 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.678573 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.679765 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.680180 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.680305 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:25.680314 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:25.680509 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 08:19:25.680547 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 08:19:25.680704 12795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:25.677126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:41127 (local address 127.4.151.130:45151)
0619 08:19:25.677335 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:25.677345 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:25.677502 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:25.677700 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:25.677701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:25.677703 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:25.677812 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:25.677814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.678660 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.678661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:25.679607 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:25.679610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.680323 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.680324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:25.680408 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:25.680413 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:19:25.680486 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 08:19:25.680527 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":3,"thread_start_us":116,"threads_started":1}
I20260619 08:19:25.681093 12393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e1556be58ab41148e5f129b2d5a4d82" candidate_uuid: "0174ead94fee4075b70c3cde1ea725be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2b5b8712524982b806e6f1686dcb2a" is_pre_election: true
I20260619 08:19:25.681174 12673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e1556be58ab41148e5f129b2d5a4d82" candidate_uuid: "0174ead94fee4075b70c3cde1ea725be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" is_pre_election: true
I20260619 08:19:25.681238 12393 raft_consensus.cc:2454] T 8e1556be58ab41148e5f129b2d5a4d82 P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0174ead94fee4075b70c3cde1ea725be in term 0.
I20260619 08:19:25.681283 12673 raft_consensus.cc:2454] T 8e1556be58ab41148e5f129b2d5a4d82 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0174ead94fee4075b70c3cde1ea725be in term 0.
I20260619 08:19:25.681453 12467 leader_election.cc:302] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0174ead94fee4075b70c3cde1ea725be, 5f5e8cd5233e492f86f8e30c5f287be6; no voters:
I20260619 08:19:25.681608 12765 raft_consensus.cc:2781] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:19:25.681651 12765 raft_consensus.cc:482] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:19:25.681677 12765 raft_consensus.cc:3037] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:25.682323 12765 raft_consensus.cc:504] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.682435 12765 leader_election.cc:288] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be [CANDIDATE]: Term 1 election: Requested vote from peers 3e2b5b8712524982b806e6f1686dcb2a (127.4.151.129:44231), 5f5e8cd5233e492f86f8e30c5f287be6 (127.4.151.131:41127)
I20260619 08:19:25.682632 12673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e1556be58ab41148e5f129b2d5a4d82" candidate_uuid: "0174ead94fee4075b70c3cde1ea725be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f5e8cd5233e492f86f8e30c5f287be6"
I20260619 08:19:25.682660 12393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e1556be58ab41148e5f129b2d5a4d82" candidate_uuid: "0174ead94fee4075b70c3cde1ea725be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2b5b8712524982b806e6f1686dcb2a"
I20260619 08:19:25.682708 12673 raft_consensus.cc:3037] T 8e1556be58ab41148e5f129b2d5a4d82 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:25.682724 12393 raft_consensus.cc:3037] T 8e1556be58ab41148e5f129b2d5a4d82 P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:25.683452 12673 raft_consensus.cc:2454] T 8e1556be58ab41148e5f129b2d5a4d82 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0174ead94fee4075b70c3cde1ea725be in term 1.
I20260619 08:19:25.683471 12393 raft_consensus.cc:2454] T 8e1556be58ab41148e5f129b2d5a4d82 P 3e2b5b8712524982b806e6f1686dcb2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0174ead94fee4075b70c3cde1ea725be in term 1.
I20260619 08:19:25.683657 12467 leader_election.cc:302] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0174ead94fee4075b70c3cde1ea725be, 5f5e8cd5233e492f86f8e30c5f287be6; no voters:
I20260619 08:19:25.683748 12765 raft_consensus.cc:2781] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:25.683899 12765 raft_consensus.cc:686] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be [term 1 LEADER]: Becoming Leader. State: Replica: 0174ead94fee4075b70c3cde1ea725be, State: Running, Role: LEADER
I20260619 08:19:25.683960 12765 consensus_queue.cc:237] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.684288 12797 tablet_replica.cc:439] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be: TxnStatusTablet state changed. Reason: New leader 0174ead94fee4075b70c3cde1ea725be. Latest consensus state: current_term: 1 leader_uuid: "0174ead94fee4075b70c3cde1ea725be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } } }
I20260619 08:19:25.684407 12797 tablet_replica.cc:442] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be: This TxnStatusTablet replica's current role is: LEADER
I20260619 08:19:25.684584 12799 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 08:19:25.684691 12799 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 08:19:25.684628 12235 catalog_manager.cc:5495] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be reported cstate change: term changed from 0 to 1, leader changed from <none> to 0174ead94fee4075b70c3cde1ea725be (127.4.151.130). New cstate: current_term: 1 leader_uuid: "0174ead94fee4075b70c3cde1ea725be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } health_report { overall_health: UNKNOWN } } }
I20260619 08:19:25.695220 12744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:25.692626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43630 (local address 127.4.151.130:36549)
0619 08:19:25.692687 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:25.692690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:25.692722 (+ 32us) server_negotiation.cc:407] Connection header received
0619 08:19:25.692832 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:25.692833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:25.692851 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:25.692930 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:19:25.693065 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.693597 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.694381 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.694569 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.694676 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:25.694863 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:19:25.694960 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:19:25.695071 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:25.695130 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 08:19:25.695149 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 08:19:25.695274 12790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:25.692548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:36549 (local address 127.0.0.1:43630)
0619 08:19:25.692675 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:25.692687 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:25.692742 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:25.692936 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:25.692937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:25.692938 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:19:25.692988 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:25.692990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.693675 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.693676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:25.694281 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:25.694283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.694675 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.694676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:25.694740 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:25.694760 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:19:25.695055 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:19:25.695064 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 08:19:25.695124 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 08:19:25.695174 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 08:19:25.696475 12673 raft_consensus.cc:1261] T 8e1556be58ab41148e5f129b2d5a4d82 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 1 FOLLOWER]: Refusing update from remote peer 0174ead94fee4075b70c3cde1ea725be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:19:25.696563 12393 raft_consensus.cc:1261] T 8e1556be58ab41148e5f129b2d5a4d82 P 3e2b5b8712524982b806e6f1686dcb2a [term 1 FOLLOWER]: Refusing update from remote peer 0174ead94fee4075b70c3cde1ea725be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:19:25.696790 12797 consensus_queue.cc:1035] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:19:25.696893 12765 consensus_queue.cc:1035] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:19:25.697474 12764 tablet_replica.cc:439] T 8e1556be58ab41148e5f129b2d5a4d82 P 5f5e8cd5233e492f86f8e30c5f287be6: TxnStatusTablet state changed. Reason: New leader 0174ead94fee4075b70c3cde1ea725be. Latest consensus state: current_term: 1 leader_uuid: "0174ead94fee4075b70c3cde1ea725be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } } }
I20260619 08:19:25.697577 12764 tablet_replica.cc:442] T 8e1556be58ab41148e5f129b2d5a4d82 P 5f5e8cd5233e492f86f8e30c5f287be6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:19:25.698035 12758 tablet_replica.cc:439] T 8e1556be58ab41148e5f129b2d5a4d82 P 3e2b5b8712524982b806e6f1686dcb2a: TxnStatusTablet state changed. Reason: New leader 0174ead94fee4075b70c3cde1ea725be. Latest consensus state: current_term: 1 leader_uuid: "0174ead94fee4075b70c3cde1ea725be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } } }
I20260619 08:19:25.698138 12758 tablet_replica.cc:442] T 8e1556be58ab41148e5f129b2d5a4d82 P 3e2b5b8712524982b806e6f1686dcb2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:19:25.698519 12802 tablet_replica.cc:439] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0174ead94fee4075b70c3cde1ea725be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } } }
I20260619 08:19:25.698622 12802 tablet_replica.cc:442] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be: This TxnStatusTablet replica's current role is: LEADER
I20260619 08:19:25.698673 12802 tablet_replica.cc:439] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0174ead94fee4075b70c3cde1ea725be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } } }
I20260619 08:19:25.698712 12802 tablet_replica.cc:442] T 8e1556be58ab41148e5f129b2d5a4d82 P 0174ead94fee4075b70c3cde1ea725be: This TxnStatusTablet replica's current role is: LEADER
I20260619 08:19:25.699074 12758 tablet_replica.cc:439] T 8e1556be58ab41148e5f129b2d5a4d82 P 3e2b5b8712524982b806e6f1686dcb2a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0174ead94fee4075b70c3cde1ea725be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } } }
I20260619 08:19:25.699167 12758 tablet_replica.cc:442] T 8e1556be58ab41148e5f129b2d5a4d82 P 3e2b5b8712524982b806e6f1686dcb2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:19:25.699365 12764 tablet_replica.cc:439] T 8e1556be58ab41148e5f129b2d5a4d82 P 5f5e8cd5233e492f86f8e30c5f287be6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0174ead94fee4075b70c3cde1ea725be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } } }
I20260619 08:19:25.699451 12764 tablet_replica.cc:442] T 8e1556be58ab41148e5f129b2d5a4d82 P 5f5e8cd5233e492f86f8e30c5f287be6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:19:25.705439 12744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:25.702409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:48691 (local address 127.4.151.130:36549)
0619 08:19:25.702446 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:25.702448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:25.702649 (+ 201us) server_negotiation.cc:407] Connection header received
0619 08:19:25.702676 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:25.702677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:25.702687 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:25.702744 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:19:25.703009 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.703519 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.704663 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.704854 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.704916 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:25.705140 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:19:25.705177 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:19:25.705206 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:25.705310 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 08:19:25.705328 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 08:19:25.705484 12808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:25.702326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:36549 (local address 127.4.151.129:48691)
0619 08:19:25.702485 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:25.702496 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:25.702559 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:25.702771 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:25.702772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:25.702774 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:19:25.702833 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:25.702835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.703614 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.703616 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:25.704561 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:25.704564 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.704939 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.704940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:25.705012 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:25.705031 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:19:25.705260 (+ 229us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:19:25.705262 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:19:25.705305 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 08:19:25.705357 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
I20260619 08:19:25.707777 12533 raft_consensus.cc:1261] T f1ae6647d3774578825a68b1b04ccf87 P 0174ead94fee4075b70c3cde1ea725be [term 1 FOLLOWER]: Refusing update from remote peer 3e2b5b8712524982b806e6f1686dcb2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:19:25.707782 12673 raft_consensus.cc:1261] T f1ae6647d3774578825a68b1b04ccf87 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 1 FOLLOWER]: Refusing update from remote peer 3e2b5b8712524982b806e6f1686dcb2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:19:25.708021 12771 consensus_queue.cc:1035] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:19:25.708115 12758 consensus_queue.cc:1035] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:19:25.716703 12762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:25.713948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:34459 (local address 127.4.151.129:44231)
0619 08:19:25.713986 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:25.713988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:25.714147 (+ 159us) server_negotiation.cc:407] Connection header received
0619 08:19:25.714264 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:25.714265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:25.714277 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:25.714330 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:19:25.714519 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.715125 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.715911 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.716130 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.716212 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:25.716427 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:19:25.716461 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:19:25.716513 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:25.716586 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 08:19:25.716613 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:19:25.716724 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:25.713885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:44231 (local address 127.4.151.130:34459)
0619 08:19:25.714056 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:25.714078 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:25.714176 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:25.714373 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:25.714374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:25.714376 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:19:25.714449 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:25.714451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.715217 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.715219 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:25.715825 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:25.715827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.716263 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.716264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:25.716328 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:25.716342 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:19:25.716518 (+ 176us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:19:25.716521 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:19:25.716565 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 08:19:25.716622 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":108,"threads_started":1}
I20260619 08:19:25.723165 12234 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:19:25.723304 12234 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:19:25.723991 12234 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8: Sending DeleteTablet for 3 replicas of tablet f1ae6647d3774578825a68b1b04ccf87
I20260619 08:19:25.724270 12653 tablet_service.cc:1514] Processing DeleteTablet for tablet f1ae6647d3774578825a68b1b04ccf87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:25 UTC) from {username='oryx', principal='oryx/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:49872
I20260619 08:19:25.724366 12775 tablet_replica.cc:330] T f1ae6647d3774578825a68b1b04ccf87 P 5f5e8cd5233e492f86f8e30c5f287be6: stopping tablet replica
I20260619 08:19:25.724412 12775 raft_consensus.cc:2229] T f1ae6647d3774578825a68b1b04ccf87 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:19:25.724449 12775 raft_consensus.cc:2258] T f1ae6647d3774578825a68b1b04ccf87 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:25.724517 12373 tablet_service.cc:1514] Processing DeleteTablet for tablet f1ae6647d3774578825a68b1b04ccf87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:25 UTC) from {username='oryx', principal='oryx/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:45296
I20260619 08:19:25.724606 12777 tablet_replica.cc:330] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a: stopping tablet replica
I20260619 08:19:25.724660 12777 raft_consensus.cc:2229] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:19:25.724710 12777 raft_consensus.cc:2258] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:25.726503 12777 ts_tablet_manager.cc:1905] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:25.728117 12513 tablet_service.cc:1514] Processing DeleteTablet for tablet f1ae6647d3774578825a68b1b04ccf87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:25 UTC) from {username='oryx', principal='oryx/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:43624
I20260619 08:19:25.728283 12776 tablet_replica.cc:330] T f1ae6647d3774578825a68b1b04ccf87 P 0174ead94fee4075b70c3cde1ea725be: stopping tablet replica
I20260619 08:19:25.728396 12776 raft_consensus.cc:2229] T f1ae6647d3774578825a68b1b04ccf87 P 0174ead94fee4075b70c3cde1ea725be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:19:25.728453 12776 raft_consensus.cc:2258] T f1ae6647d3774578825a68b1b04ccf87 P 0174ead94fee4075b70c3cde1ea725be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:25.728668 12777 ts_tablet_manager.cc:1918] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:19:25.728735 12777 log.cc:1196] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-0/wal/wals/f1ae6647d3774578825a68b1b04ccf87
I20260619 08:19:25.728753 12775 ts_tablet_manager.cc:1905] T f1ae6647d3774578825a68b1b04ccf87 P 5f5e8cd5233e492f86f8e30c5f287be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:25.728856 12776 ts_tablet_manager.cc:1905] T f1ae6647d3774578825a68b1b04ccf87 P 0174ead94fee4075b70c3cde1ea725be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:25.728964 12777 ts_tablet_manager.cc:1939] T f1ae6647d3774578825a68b1b04ccf87 P 3e2b5b8712524982b806e6f1686dcb2a: Deleting consensus metadata
I20260619 08:19:25.729300 12222 catalog_manager.cc:4841] TS 3e2b5b8712524982b806e6f1686dcb2a (127.4.151.129:44231): tablet f1ae6647d3774578825a68b1b04ccf87 (table test-table [id=775cab1044ca4c668c890ad325b8e4d4]) successfully deleted
I20260619 08:19:25.730438 12776 ts_tablet_manager.cc:1918] T f1ae6647d3774578825a68b1b04ccf87 P 0174ead94fee4075b70c3cde1ea725be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:19:25.730465 12775 ts_tablet_manager.cc:1918] T f1ae6647d3774578825a68b1b04ccf87 P 5f5e8cd5233e492f86f8e30c5f287be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:19:25.730517 12775 log.cc:1196] T f1ae6647d3774578825a68b1b04ccf87 P 5f5e8cd5233e492f86f8e30c5f287be6: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/f1ae6647d3774578825a68b1b04ccf87
I20260619 08:19:25.730517 12776 log.cc:1196] T f1ae6647d3774578825a68b1b04ccf87 P 0174ead94fee4075b70c3cde1ea725be: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/f1ae6647d3774578825a68b1b04ccf87
I20260619 08:19:25.730751 12776 ts_tablet_manager.cc:1939] T f1ae6647d3774578825a68b1b04ccf87 P 0174ead94fee4075b70c3cde1ea725be: Deleting consensus metadata
I20260619 08:19:25.730791 12775 ts_tablet_manager.cc:1939] T f1ae6647d3774578825a68b1b04ccf87 P 5f5e8cd5233e492f86f8e30c5f287be6: Deleting consensus metadata
I20260619 08:19:25.731411 12223 catalog_manager.cc:4841] TS 0174ead94fee4075b70c3cde1ea725be (127.4.151.130:36549): tablet f1ae6647d3774578825a68b1b04ccf87 (table test-table [id=775cab1044ca4c668c890ad325b8e4d4]) successfully deleted
I20260619 08:19:25.731492 12223 catalog_manager.cc:4841] TS 5f5e8cd5233e492f86f8e30c5f287be6 (127.4.151.131:41127): tablet f1ae6647d3774578825a68b1b04ccf87 (table test-table [id=775cab1044ca4c668c890ad325b8e4d4]) successfully deleted
Jun 19 08:19:25 dist-test-slave-d3n8 krb5kdc[12191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 08:19:25 dist-test-slave-d3n8 krb5kdc[12191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:25 dist-test-slave-d3n8 krb5kdc[12191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
W20260619 08:19:25.752427 12791 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:25.746451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52978 (local address 127.4.151.190:38339)
0619 08:19:25.746521 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:25.746523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:25.746634 (+ 111us) server_negotiation.cc:407] Connection header received
0619 08:19:25.746672 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:25.746674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:25.746687 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:25.746762 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:25.747307 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.748112 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.748866 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.749201 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.749276 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:25.752334 (+ 3058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52978: BlockingRecv error: recv got EOF from 127.0.0.1:52978 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 19 08:19:25 dist-test-slave-d3n8 krb5kdc[12191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM
I20260619 08:19:25.761857 12791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:25.753532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52990 (local address 127.4.151.190:38339)
0619 08:19:25.753582 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:25.753584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:25.753598 (+ 14us) server_negotiation.cc:407] Connection header received
0619 08:19:25.753685 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:25.753686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:25.753697 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:25.753777 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:25.754360 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.754952 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.755746 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.755980 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.756084 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:25.758899 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:25.758900 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:25.758900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:25.758908 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:25.760528 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:25.760953 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:25.760954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:25.760954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:25.761017 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:25.761219 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:25.761221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:25.761221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:25.761340 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:25.761517 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:25.761652 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 08:19:25.761739 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 08:19:25.763576 12234 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:19:25.763756 12234 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:19:25.765298 12373 tablet_service.cc:1467] Processing CreateTablet for tablet 761be253499c45fcaf8c3e735f33d6b4 (DEFAULT_TABLE table=test-table [id=573fed70c785490abed7d5274d8f0754]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:25.765298 12513 tablet_service.cc:1467] Processing CreateTablet for tablet 761be253499c45fcaf8c3e735f33d6b4 (DEFAULT_TABLE table=test-table [id=573fed70c785490abed7d5274d8f0754]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:25.765486 12513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 761be253499c45fcaf8c3e735f33d6b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:25.765486 12373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 761be253499c45fcaf8c3e735f33d6b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:25.766852 12778 tablet_bootstrap.cc:492] T 761be253499c45fcaf8c3e735f33d6b4 P 0174ead94fee4075b70c3cde1ea725be: Bootstrap starting.
I20260619 08:19:25.766884 12653 tablet_service.cc:1467] Processing CreateTablet for tablet 761be253499c45fcaf8c3e735f33d6b4 (DEFAULT_TABLE table=test-table [id=573fed70c785490abed7d5274d8f0754]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:25.767009 12653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 761be253499c45fcaf8c3e735f33d6b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:25.767257 12778 tablet_bootstrap.cc:654] T 761be253499c45fcaf8c3e735f33d6b4 P 0174ead94fee4075b70c3cde1ea725be: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:25.767874 12778 tablet_bootstrap.cc:492] T 761be253499c45fcaf8c3e735f33d6b4 P 0174ead94fee4075b70c3cde1ea725be: No bootstrap required, opened a new log
I20260619 08:19:25.767936 12778 ts_tablet_manager.cc:1397] T 761be253499c45fcaf8c3e735f33d6b4 P 0174ead94fee4075b70c3cde1ea725be: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:25.768052 12778 raft_consensus.cc:348] T 761be253499c45fcaf8c3e735f33d6b4 P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.768186 12778 raft_consensus.cc:374] T 761be253499c45fcaf8c3e735f33d6b4 P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:25.768275 12778 raft_consensus.cc:729] T 761be253499c45fcaf8c3e735f33d6b4 P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0174ead94fee4075b70c3cde1ea725be, State: Initialized, Role: FOLLOWER
I20260619 08:19:25.768347 12779 tablet_bootstrap.cc:492] T 761be253499c45fcaf8c3e735f33d6b4 P 3e2b5b8712524982b806e6f1686dcb2a: Bootstrap starting.
I20260619 08:19:25.768327 12778 consensus_queue.cc:260] T 761be253499c45fcaf8c3e735f33d6b4 P 0174ead94fee4075b70c3cde1ea725be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.768453 12778 ts_tablet_manager.cc:1428] T 761be253499c45fcaf8c3e735f33d6b4 P 0174ead94fee4075b70c3cde1ea725be: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:25.768695 12779 tablet_bootstrap.cc:654] T 761be253499c45fcaf8c3e735f33d6b4 P 3e2b5b8712524982b806e6f1686dcb2a: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:25.768992 12781 tablet_bootstrap.cc:492] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6: Bootstrap starting.
I20260619 08:19:25.769356 12781 tablet_bootstrap.cc:654] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:25.769879 12779 tablet_bootstrap.cc:492] T 761be253499c45fcaf8c3e735f33d6b4 P 3e2b5b8712524982b806e6f1686dcb2a: No bootstrap required, opened a new log
I20260619 08:19:25.769935 12779 ts_tablet_manager.cc:1397] T 761be253499c45fcaf8c3e735f33d6b4 P 3e2b5b8712524982b806e6f1686dcb2a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:19:25.770057 12779 raft_consensus.cc:348] T 761be253499c45fcaf8c3e735f33d6b4 P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.770133 12779 raft_consensus.cc:374] T 761be253499c45fcaf8c3e735f33d6b4 P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:25.770147 12779 raft_consensus.cc:729] T 761be253499c45fcaf8c3e735f33d6b4 P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2b5b8712524982b806e6f1686dcb2a, State: Initialized, Role: FOLLOWER
I20260619 08:19:25.770205 12779 consensus_queue.cc:260] T 761be253499c45fcaf8c3e735f33d6b4 P 3e2b5b8712524982b806e6f1686dcb2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.770305 12779 ts_tablet_manager.cc:1428] T 761be253499c45fcaf8c3e735f33d6b4 P 3e2b5b8712524982b806e6f1686dcb2a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:25.770776 12781 tablet_bootstrap.cc:492] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6: No bootstrap required, opened a new log
I20260619 08:19:25.770828 12781 ts_tablet_manager.cc:1397] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:19:25.770968 12781 raft_consensus.cc:348] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.771049 12781 raft_consensus.cc:374] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:25.771071 12781 raft_consensus.cc:729] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f5e8cd5233e492f86f8e30c5f287be6, State: Initialized, Role: FOLLOWER
I20260619 08:19:25.771109 12781 consensus_queue.cc:260] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.771198 12781 ts_tablet_manager.cc:1428] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:25.869475 12764 raft_consensus.cc:482] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:19:25.869570 12764 raft_consensus.cc:504] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.869891 12764 leader_election.cc:288] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e2b5b8712524982b806e6f1686dcb2a (127.4.151.129:44231), 0174ead94fee4075b70c3cde1ea725be (127.4.151.130:36549)
I20260619 08:19:25.873427 12762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:25.870256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:55571 (local address 127.4.151.129:44231)
0619 08:19:25.870307 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:25.870310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:25.870444 (+ 134us) server_negotiation.cc:407] Connection header received
0619 08:19:25.870528 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:25.870529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:25.870540 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:25.870598 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:25.870787 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.871365 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.872524 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.873054 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.873111 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:25.873118 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:25.873287 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 08:19:25.873323 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 08:19:25.873482 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:25.870187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:44231 (local address 127.4.151.131:55571)
0619 08:19:25.870360 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:25.870370 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:25.870441 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:25.870620 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:25.870621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:25.870627 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:25.870714 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:25.870716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.871481 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.871482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:25.872391 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:25.872394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.873134 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.873135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:25.873206 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:25.873212 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 08:19:25.873274 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 08:19:25.873315 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":102,"threads_started":1}
I20260619 08:19:25.873704 12393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761be253499c45fcaf8c3e735f33d6b4" candidate_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2b5b8712524982b806e6f1686dcb2a" is_pre_election: true
I20260619 08:19:25.873804 12393 raft_consensus.cc:2454] T 761be253499c45fcaf8c3e735f33d6b4 P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f5e8cd5233e492f86f8e30c5f287be6 in term 0.
I20260619 08:19:25.873958 12829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:25.870100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:36549 (local address 127.4.151.131:47121)
0619 08:19:25.870274 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:25.870286 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:25.870352 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:25.870549 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:25.870550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:25.870552 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:25.870635 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:25.870637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.871361 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.871362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:25.872829 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:25.872832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.873666 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.873667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:25.873732 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:25.873735 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:19:25.873786 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 08:19:25.873822 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":77,"threads_started":1}
I20260619 08:19:25.874007 12606 leader_election.cc:302] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e2b5b8712524982b806e6f1686dcb2a, 5f5e8cd5233e492f86f8e30c5f287be6; no voters:
I20260619 08:19:25.873955 12744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:25.870146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:47121 (local address 127.4.151.130:36549)
0619 08:19:25.870228 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:25.870230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:25.870344 (+ 114us) server_negotiation.cc:407] Connection header received
0619 08:19:25.870437 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:25.870438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:25.870452 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:25.870544 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:25.870721 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.871281 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.872995 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.873557 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.873629 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:25.873638 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:25.873824 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 08:19:25.873881 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 08:19:25.874146 12764 raft_consensus.cc:2781] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:19:25.874186 12764 raft_consensus.cc:482] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:19:25.874212 12764 raft_consensus.cc:3037] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:25.874298 12533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761be253499c45fcaf8c3e735f33d6b4" candidate_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0174ead94fee4075b70c3cde1ea725be" is_pre_election: true
I20260619 08:19:25.874402 12533 raft_consensus.cc:2454] T 761be253499c45fcaf8c3e735f33d6b4 P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f5e8cd5233e492f86f8e30c5f287be6 in term 0.
I20260619 08:19:25.875051 12764 raft_consensus.cc:504] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.875195 12764 leader_election.cc:288] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6 [CANDIDATE]: Term 1 election: Requested vote from peers 3e2b5b8712524982b806e6f1686dcb2a (127.4.151.129:44231), 0174ead94fee4075b70c3cde1ea725be (127.4.151.130:36549)
I20260619 08:19:25.875339 12393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761be253499c45fcaf8c3e735f33d6b4" candidate_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2b5b8712524982b806e6f1686dcb2a"
I20260619 08:19:25.875419 12393 raft_consensus.cc:3037] T 761be253499c45fcaf8c3e735f33d6b4 P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:25.875406 12533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761be253499c45fcaf8c3e735f33d6b4" candidate_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0174ead94fee4075b70c3cde1ea725be"
I20260619 08:19:25.875476 12533 raft_consensus.cc:3037] T 761be253499c45fcaf8c3e735f33d6b4 P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:25.876070 12533 raft_consensus.cc:2454] T 761be253499c45fcaf8c3e735f33d6b4 P 0174ead94fee4075b70c3cde1ea725be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f5e8cd5233e492f86f8e30c5f287be6 in term 1.
I20260619 08:19:25.876220 12393 raft_consensus.cc:2454] T 761be253499c45fcaf8c3e735f33d6b4 P 3e2b5b8712524982b806e6f1686dcb2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f5e8cd5233e492f86f8e30c5f287be6 in term 1.
I20260619 08:19:25.876248 12607 leader_election.cc:302] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0174ead94fee4075b70c3cde1ea725be, 5f5e8cd5233e492f86f8e30c5f287be6; no voters:
I20260619 08:19:25.876343 12764 raft_consensus.cc:2781] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:25.876498 12764 raft_consensus.cc:686] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f5e8cd5233e492f86f8e30c5f287be6, State: Running, Role: LEADER
I20260619 08:19:25.876591 12764 consensus_queue.cc:237] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.877329 12235 catalog_manager.cc:5495] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f5e8cd5233e492f86f8e30c5f287be6 (127.4.151.131). New cstate: current_term: 1 leader_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } health_report { overall_health: HEALTHY } } }
I20260619 08:19:25.890654 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:25.888165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49888 (local address 127.4.151.131:41127)
0619 08:19:25.888215 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:25.888217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:25.888230 (+ 13us) server_negotiation.cc:407] Connection header received
0619 08:19:25.888313 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:25.888313 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:25.888334 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:25.888396 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:19:25.888519 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.889003 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.889830 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:25.890026 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:25.890088 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:25.890224 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:19:25.890343 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:19:25.890399 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:25.890492 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 08:19:25.890510 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:19:25.891780 12393 raft_consensus.cc:1261] T 761be253499c45fcaf8c3e735f33d6b4 P 3e2b5b8712524982b806e6f1686dcb2a [term 1 FOLLOWER]: Refusing update from remote peer 5f5e8cd5233e492f86f8e30c5f287be6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:19:25.891783 12533 raft_consensus.cc:1261] T 761be253499c45fcaf8c3e735f33d6b4 P 0174ead94fee4075b70c3cde1ea725be [term 1 FOLLOWER]: Refusing update from remote peer 5f5e8cd5233e492f86f8e30c5f287be6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:19:25.892107 12831 consensus_queue.cc:1035] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:19:25.892256 12764 consensus_queue.cc:1035] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:19:25.895824 12235 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:19:25.895913 12235 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:19:25.896746 12235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8: Sending DeleteTablet for 3 replicas of tablet 761be253499c45fcaf8c3e735f33d6b4
I20260619 08:19:25.896957 12513 tablet_service.cc:1514] Processing DeleteTablet for tablet 761be253499c45fcaf8c3e735f33d6b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:25 UTC) from {username='oryx', principal='oryx/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:43624
I20260619 08:19:25.897027 12512 tablet_service.cc:1514] Processing DeleteTablet for tablet 761be253499c45fcaf8c3e735f33d6b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:25 UTC) from {username='oryx', principal='oryx/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:43624
I20260619 08:19:25.897039 12373 tablet_service.cc:1514] Processing DeleteTablet for tablet 761be253499c45fcaf8c3e735f33d6b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:25 UTC) from {username='oryx', principal='oryx/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:45296
I20260619 08:19:25.897142 12776 tablet_replica.cc:330] T 761be253499c45fcaf8c3e735f33d6b4 P 0174ead94fee4075b70c3cde1ea725be: stopping tablet replica
I20260619 08:19:25.897217 12776 raft_consensus.cc:2229] T 761be253499c45fcaf8c3e735f33d6b4 P 0174ead94fee4075b70c3cde1ea725be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:19:25.897209 12653 tablet_service.cc:1514] Processing DeleteTablet for tablet 761be253499c45fcaf8c3e735f33d6b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:25 UTC) from {username='oryx', principal='oryx/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:49872
I20260619 08:19:25.897274 12776 raft_consensus.cc:2258] T 761be253499c45fcaf8c3e735f33d6b4 P 0174ead94fee4075b70c3cde1ea725be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:25.897294 12775 tablet_replica.cc:330] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6: stopping tablet replica
I20260619 08:19:25.897349 12775 raft_consensus.cc:2229] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:19:25.897362 12777 tablet_replica.cc:330] T 761be253499c45fcaf8c3e735f33d6b4 P 3e2b5b8712524982b806e6f1686dcb2a: stopping tablet replica
I20260619 08:19:25.897447 12777 raft_consensus.cc:2229] T 761be253499c45fcaf8c3e735f33d6b4 P 3e2b5b8712524982b806e6f1686dcb2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:19:25.897466 12775 raft_consensus.cc:2258] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:25.897517 12777 raft_consensus.cc:2258] T 761be253499c45fcaf8c3e735f33d6b4 P 3e2b5b8712524982b806e6f1686dcb2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:25.897794 12775 ts_tablet_manager.cc:1905] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:25.897971 12776 ts_tablet_manager.cc:1905] T 761be253499c45fcaf8c3e735f33d6b4 P 0174ead94fee4075b70c3cde1ea725be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:25.899498 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 08:19:25.899606 12776 ts_tablet_manager.cc:1918] T 761be253499c45fcaf8c3e735f33d6b4 P 0174ead94fee4075b70c3cde1ea725be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:19:25.899607 12775 ts_tablet_manager.cc:1918] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:19:25.899662 12775 log.cc:1196] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/761be253499c45fcaf8c3e735f33d6b4
I20260619 08:19:25.899662 12776 log.cc:1196] T 761be253499c45fcaf8c3e735f33d6b4 P 0174ead94fee4075b70c3cde1ea725be: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/761be253499c45fcaf8c3e735f33d6b4
W20260619 08:19:25.899669 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:19:25.899883 12776 ts_tablet_manager.cc:1939] T 761be253499c45fcaf8c3e735f33d6b4 P 0174ead94fee4075b70c3cde1ea725be: Deleting consensus metadata
I20260619 08:19:25.899894 12775 ts_tablet_manager.cc:1939] T 761be253499c45fcaf8c3e735f33d6b4 P 5f5e8cd5233e492f86f8e30c5f287be6: Deleting consensus metadata
I20260619 08:19:25.900259 12777 ts_tablet_manager.cc:1905] T 761be253499c45fcaf8c3e735f33d6b4 P 3e2b5b8712524982b806e6f1686dcb2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:25.900319 12223 catalog_manager.cc:4841] TS 5f5e8cd5233e492f86f8e30c5f287be6 (127.4.151.131:41127): tablet 761be253499c45fcaf8c3e735f33d6b4 (table test-table [id=573fed70c785490abed7d5274d8f0754]) successfully deleted
I20260619 08:19:25.901240 12223 catalog_manager.cc:4841] TS 0174ead94fee4075b70c3cde1ea725be (127.4.151.130:36549): tablet 761be253499c45fcaf8c3e735f33d6b4 (table test-table [id=573fed70c785490abed7d5274d8f0754]) successfully deleted
W20260619 08:19:25.901412 12223 catalog_manager.cc:4808] TS 0174ead94fee4075b70c3cde1ea725be (127.4.151.130:36549): delete failed for tablet 761be253499c45fcaf8c3e735f33d6b4 because the tablet was not found. No further retry: Not found: Tablet not found: 761be253499c45fcaf8c3e735f33d6b4
I20260619 08:19:25.901921 12777 ts_tablet_manager.cc:1918] T 761be253499c45fcaf8c3e735f33d6b4 P 3e2b5b8712524982b806e6f1686dcb2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:19:25.901993 12777 log.cc:1196] T 761be253499c45fcaf8c3e735f33d6b4 P 3e2b5b8712524982b806e6f1686dcb2a: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-0/wal/wals/761be253499c45fcaf8c3e735f33d6b4
I20260619 08:19:25.902228 12777 ts_tablet_manager.cc:1939] T 761be253499c45fcaf8c3e735f33d6b4 P 3e2b5b8712524982b806e6f1686dcb2a: Deleting consensus metadata
I20260619 08:19:25.902601 12222 catalog_manager.cc:4841] TS 3e2b5b8712524982b806e6f1686dcb2a (127.4.151.129:44231): tablet 761be253499c45fcaf8c3e735f33d6b4 (table test-table [id=573fed70c785490abed7d5274d8f0754]) successfully deleted
I20260619 08:19:25.903359 12373 tablet_service.cc:1467] Processing CreateTablet for tablet d84e20d59640407cb4d900171a796f94 (DEFAULT_TABLE table=test-table [id=a210cbe8b0664172baf7e3ae98abc8c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:25.903447 12653 tablet_service.cc:1467] Processing CreateTablet for tablet d84e20d59640407cb4d900171a796f94 (DEFAULT_TABLE table=test-table [id=a210cbe8b0664172baf7e3ae98abc8c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:25.903502 12373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84e20d59640407cb4d900171a796f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:25.903584 12653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84e20d59640407cb4d900171a796f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:25.904428 12779 tablet_bootstrap.cc:492] T d84e20d59640407cb4d900171a796f94 P 3e2b5b8712524982b806e6f1686dcb2a: Bootstrap starting.
I20260619 08:19:25.904558 12512 tablet_service.cc:1467] Processing CreateTablet for tablet d84e20d59640407cb4d900171a796f94 (DEFAULT_TABLE table=test-table [id=a210cbe8b0664172baf7e3ae98abc8c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:25.904702 12512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84e20d59640407cb4d900171a796f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:25.904763 12779 tablet_bootstrap.cc:654] T d84e20d59640407cb4d900171a796f94 P 3e2b5b8712524982b806e6f1686dcb2a: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:25.904902 12781 tablet_bootstrap.cc:492] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6: Bootstrap starting.
I20260619 08:19:25.905313 12781 tablet_bootstrap.cc:654] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:25.905967 12781 tablet_bootstrap.cc:492] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6: No bootstrap required, opened a new log
I20260619 08:19:25.906035 12781 ts_tablet_manager.cc:1397] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:25.906090 12778 tablet_bootstrap.cc:492] T d84e20d59640407cb4d900171a796f94 P 0174ead94fee4075b70c3cde1ea725be: Bootstrap starting.
I20260619 08:19:25.906200 12781 raft_consensus.cc:348] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.906291 12781 raft_consensus.cc:374] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:25.906319 12781 raft_consensus.cc:729] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f5e8cd5233e492f86f8e30c5f287be6, State: Initialized, Role: FOLLOWER
I20260619 08:19:25.906363 12781 consensus_queue.cc:260] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.906416 12778 tablet_bootstrap.cc:654] T d84e20d59640407cb4d900171a796f94 P 0174ead94fee4075b70c3cde1ea725be: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:25.906454 12781 ts_tablet_manager.cc:1428] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:25.906955 12778 tablet_bootstrap.cc:492] T d84e20d59640407cb4d900171a796f94 P 0174ead94fee4075b70c3cde1ea725be: No bootstrap required, opened a new log
I20260619 08:19:25.907011 12778 ts_tablet_manager.cc:1397] T d84e20d59640407cb4d900171a796f94 P 0174ead94fee4075b70c3cde1ea725be: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:25.907126 12778 raft_consensus.cc:348] T d84e20d59640407cb4d900171a796f94 P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.907196 12778 raft_consensus.cc:374] T d84e20d59640407cb4d900171a796f94 P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:25.907217 12778 raft_consensus.cc:729] T d84e20d59640407cb4d900171a796f94 P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0174ead94fee4075b70c3cde1ea725be, State: Initialized, Role: FOLLOWER
I20260619 08:19:25.907262 12778 consensus_queue.cc:260] T d84e20d59640407cb4d900171a796f94 P 0174ead94fee4075b70c3cde1ea725be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.907358 12778 ts_tablet_manager.cc:1428] T d84e20d59640407cb4d900171a796f94 P 0174ead94fee4075b70c3cde1ea725be: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 08:19:25.913921 12779 tablet_bootstrap.cc:492] T d84e20d59640407cb4d900171a796f94 P 3e2b5b8712524982b806e6f1686dcb2a: No bootstrap required, opened a new log
I20260619 08:19:25.914008 12779 ts_tablet_manager.cc:1397] T d84e20d59640407cb4d900171a796f94 P 3e2b5b8712524982b806e6f1686dcb2a: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260619 08:19:25.914145 12779 raft_consensus.cc:348] T d84e20d59640407cb4d900171a796f94 P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.914280 12779 raft_consensus.cc:374] T d84e20d59640407cb4d900171a796f94 P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:25.914314 12779 raft_consensus.cc:729] T d84e20d59640407cb4d900171a796f94 P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2b5b8712524982b806e6f1686dcb2a, State: Initialized, Role: FOLLOWER
I20260619 08:19:25.914367 12779 consensus_queue.cc:260] T d84e20d59640407cb4d900171a796f94 P 3e2b5b8712524982b806e6f1686dcb2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:25.914470 12779 ts_tablet_manager.cc:1428] T d84e20d59640407cb4d900171a796f94 P 3e2b5b8712524982b806e6f1686dcb2a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:26.012195 12837 raft_consensus.cc:482] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:19:26.012571 12837 raft_consensus.cc:504] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:26.012844 12837 leader_election.cc:288] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e2b5b8712524982b806e6f1686dcb2a (127.4.151.129:44231), 0174ead94fee4075b70c3cde1ea725be (127.4.151.130:36549)
I20260619 08:19:26.013059 12533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84e20d59640407cb4d900171a796f94" candidate_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0174ead94fee4075b70c3cde1ea725be" is_pre_election: true
I20260619 08:19:26.013056 12393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84e20d59640407cb4d900171a796f94" candidate_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2b5b8712524982b806e6f1686dcb2a" is_pre_election: true
I20260619 08:19:26.013170 12393 raft_consensus.cc:2454] T d84e20d59640407cb4d900171a796f94 P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f5e8cd5233e492f86f8e30c5f287be6 in term 0.
I20260619 08:19:26.013170 12533 raft_consensus.cc:2454] T d84e20d59640407cb4d900171a796f94 P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f5e8cd5233e492f86f8e30c5f287be6 in term 0.
I20260619 08:19:26.013317 12607 leader_election.cc:302] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0174ead94fee4075b70c3cde1ea725be, 5f5e8cd5233e492f86f8e30c5f287be6; no voters:
I20260619 08:19:26.013764 12837 raft_consensus.cc:2781] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:19:26.013805 12837 raft_consensus.cc:482] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:19:26.013826 12837 raft_consensus.cc:3037] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:26.014698 12837 raft_consensus.cc:504] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:26.014866 12837 leader_election.cc:288] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6 [CANDIDATE]: Term 1 election: Requested vote from peers 3e2b5b8712524982b806e6f1686dcb2a (127.4.151.129:44231), 0174ead94fee4075b70c3cde1ea725be (127.4.151.130:36549)
I20260619 08:19:26.015061 12533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84e20d59640407cb4d900171a796f94" candidate_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0174ead94fee4075b70c3cde1ea725be"
I20260619 08:19:26.015143 12533 raft_consensus.cc:3037] T d84e20d59640407cb4d900171a796f94 P 0174ead94fee4075b70c3cde1ea725be [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:26.015872 12533 raft_consensus.cc:2454] T d84e20d59640407cb4d900171a796f94 P 0174ead94fee4075b70c3cde1ea725be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f5e8cd5233e492f86f8e30c5f287be6 in term 1.
I20260619 08:19:26.016018 12393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84e20d59640407cb4d900171a796f94" candidate_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2b5b8712524982b806e6f1686dcb2a"
I20260619 08:19:26.016108 12393 raft_consensus.cc:3037] T d84e20d59640407cb4d900171a796f94 P 3e2b5b8712524982b806e6f1686dcb2a [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:26.016077 12607 leader_election.cc:302] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0174ead94fee4075b70c3cde1ea725be, 5f5e8cd5233e492f86f8e30c5f287be6; no voters:
I20260619 08:19:26.016186 12837 raft_consensus.cc:2781] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:26.016232 12837 raft_consensus.cc:686] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f5e8cd5233e492f86f8e30c5f287be6, State: Running, Role: LEADER
I20260619 08:19:26.016279 12837 consensus_queue.cc:237] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } }
I20260619 08:19:26.016813 12235 catalog_manager.cc:5495] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f5e8cd5233e492f86f8e30c5f287be6 (127.4.151.131). New cstate: current_term: 1 leader_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f5e8cd5233e492f86f8e30c5f287be6" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 41127 } health_report { overall_health: HEALTHY } } }
I20260619 08:19:26.016906 12393 raft_consensus.cc:2454] T d84e20d59640407cb4d900171a796f94 P 3e2b5b8712524982b806e6f1686dcb2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f5e8cd5233e492f86f8e30c5f287be6 in term 1.
I20260619 08:19:26.029134 12744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:26.026407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:51031 (local address 127.4.151.130:36549)
0619 08:19:26.026454 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:26.026456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:26.026595 (+ 139us) server_negotiation.cc:407] Connection header received
0619 08:19:26.026628 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:26.026629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:26.026639 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:26.026701 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:19:26.026800 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:26.027540 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:26.028335 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:26.028531 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:26.028589 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:26.028868 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:19:26.028898 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:19:26.028963 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:26.029053 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 08:19:26.029077 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 08:19:26.029146 12844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:26.026319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:36549 (local address 127.4.151.131:51031)
0619 08:19:26.026528 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:26.026538 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:26.026590 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:26.026710 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:26.026711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:26.026712 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:19:26.026758 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:26.026760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:26.027625 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:26.027626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:26.028231 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:26.028233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:26.028631 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:26.028632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:26.028682 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:26.028697 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:19:26.028999 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:19:26.029002 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:19:26.029044 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 08:19:26.029087 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":83,"threads_started":1}
I20260619 08:19:26.031279 12533 raft_consensus.cc:1261] T d84e20d59640407cb4d900171a796f94 P 0174ead94fee4075b70c3cde1ea725be [term 1 FOLLOWER]: Refusing update from remote peer 5f5e8cd5233e492f86f8e30c5f287be6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:19:26.031307 12393 raft_consensus.cc:1261] T d84e20d59640407cb4d900171a796f94 P 3e2b5b8712524982b806e6f1686dcb2a [term 1 FOLLOWER]: Refusing update from remote peer 5f5e8cd5233e492f86f8e30c5f287be6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:19:26.031540 12831 consensus_queue.cc:1035] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e2b5b8712524982b806e6f1686dcb2a" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 44231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:19:26.031644 12837 consensus_queue.cc:1035] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0174ead94fee4075b70c3cde1ea725be" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 36549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:19:26.039255 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:26.036628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:54669 (local address 127.4.151.131:41127)
0619 08:19:26.036806 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:26.036807 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:26.036814 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:19:26.036834 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:26.036835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:26.036846 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:26.036899 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:19:26.037039 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:26.037584 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:26.038465 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:26.038749 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:26.038849 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:26.039002 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:19:26.039039 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:19:26.039096 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:26.039179 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 08:19:26.039198 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260619 08:19:26.039278 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:26.036629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:41127 (local address 127.4.151.130:54669)
0619 08:19:26.036668 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:26.036680 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:26.036733 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:26.036929 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:26.036930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:26.036932 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:19:26.036977 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:26.036978 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:26.037651 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:26.037652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:26.038216 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:26.038218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:26.038855 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:26.038856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:26.038922 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:26.038930 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:19:26.039135 (+ 205us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:19:26.039137 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:19:26.039174 (+ 37us) client_negotiation.cc:240] Negotiation successful
0619 08:19:26.039217 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260619 08:19:26.045433 12235 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:19:26.045547 12235 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:19:26.046257 12235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32506c1f476b4b3cb615f6f2ef7a8cb8: Sending DeleteTablet for 3 replicas of tablet d84e20d59640407cb4d900171a796f94
I20260619 08:19:26.046522 12512 tablet_service.cc:1514] Processing DeleteTablet for tablet d84e20d59640407cb4d900171a796f94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:26 UTC) from {username='oryx', principal='oryx/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:43624
I20260619 08:19:26.046553 12373 tablet_service.cc:1514] Processing DeleteTablet for tablet d84e20d59640407cb4d900171a796f94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:26 UTC) from {username='oryx', principal='oryx/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:45296
I20260619 08:19:26.046658 12776 tablet_replica.cc:330] T d84e20d59640407cb4d900171a796f94 P 0174ead94fee4075b70c3cde1ea725be: stopping tablet replica
I20260619 08:19:26.046669 12653 tablet_service.cc:1514] Processing DeleteTablet for tablet d84e20d59640407cb4d900171a796f94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:26 UTC) from {username='oryx', principal='oryx/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:49872
I20260619 08:19:26.046729 12776 raft_consensus.cc:2229] T d84e20d59640407cb4d900171a796f94 P 0174ead94fee4075b70c3cde1ea725be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:19:26.046753 12775 tablet_replica.cc:330] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6: stopping tablet replica
I20260619 08:19:26.046765 12776 raft_consensus.cc:2258] T d84e20d59640407cb4d900171a796f94 P 0174ead94fee4075b70c3cde1ea725be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:26.046805 12775 raft_consensus.cc:2229] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:19:26.046866 12775 raft_consensus.cc:2258] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:26.046941 12777 tablet_replica.cc:330] T d84e20d59640407cb4d900171a796f94 P 3e2b5b8712524982b806e6f1686dcb2a: stopping tablet replica
I20260619 08:19:26.047003 12777 raft_consensus.cc:2229] T d84e20d59640407cb4d900171a796f94 P 3e2b5b8712524982b806e6f1686dcb2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:19:26.047040 12777 raft_consensus.cc:2258] T d84e20d59640407cb4d900171a796f94 P 3e2b5b8712524982b806e6f1686dcb2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:26.047168 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 12308
I20260619 08:19:26.047216 12776 ts_tablet_manager.cc:1905] T d84e20d59640407cb4d900171a796f94 P 0174ead94fee4075b70c3cde1ea725be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:26.048899 12776 ts_tablet_manager.cc:1918] T d84e20d59640407cb4d900171a796f94 P 0174ead94fee4075b70c3cde1ea725be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:19:26.048960 12776 log.cc:1196] T d84e20d59640407cb4d900171a796f94 P 0174ead94fee4075b70c3cde1ea725be: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/d84e20d59640407cb4d900171a796f94
I20260619 08:19:26.049073 12775 ts_tablet_manager.cc:1905] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:26.049190 12776 ts_tablet_manager.cc:1939] T d84e20d59640407cb4d900171a796f94 P 0174ead94fee4075b70c3cde1ea725be: Deleting consensus metadata
I20260619 08:19:26.049642 12223 catalog_manager.cc:4841] TS 0174ead94fee4075b70c3cde1ea725be (127.4.151.130:36549): tablet d84e20d59640407cb4d900171a796f94 (table test-table [id=a210cbe8b0664172baf7e3ae98abc8c6]) successfully deleted
I20260619 08:19:26.050629 12775 ts_tablet_manager.cc:1918] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:19:26.050716 12775 log.cc:1196] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/d84e20d59640407cb4d900171a796f94
I20260619 08:19:26.051020 12775 ts_tablet_manager.cc:1939] T d84e20d59640407cb4d900171a796f94 P 5f5e8cd5233e492f86f8e30c5f287be6: Deleting consensus metadata
I20260619 08:19:26.051520 12223 catalog_manager.cc:4841] TS 5f5e8cd5233e492f86f8e30c5f287be6 (127.4.151.131:41127): tablet d84e20d59640407cb4d900171a796f94 (table test-table [id=a210cbe8b0664172baf7e3ae98abc8c6]) successfully deleted
W20260619 08:19:26.055020 12222 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.151.129:44231 (error 108)
I20260619 08:19:26.055291 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 12448
I20260619 08:19:26.056636 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:26.056394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:44231 (local address 127.0.0.1:45316)
0619 08:19:26.056551 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:26.056599 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.129:44231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
W20260619 08:19:26.056787 12222 catalog_manager.cc:4584] TS 3e2b5b8712524982b806e6f1686dcb2a (127.4.151.129:44231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d84e20d59640407cb4d900171a796f94: Network error: Client connection negotiation failed: client connection to 127.4.151.129:44231: connect: Connection refused (error 111)
I20260619 08:19:26.062085 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 12588
I20260619 08:19:26.069362 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 12204
2026-06-19T08:19:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3119 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:19:26 dist-test-slave-d3n8 krb5kdc[12851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:19:26 dist-test-slave-d3n8 krb5kdc[12851](info): set up 2 sockets
Jun 19 08:19:26 dist-test-slave-d3n8 krb5kdc[12851](info): commencing operation
krb5kdc: starting...
W20260619 08:19:27.154145 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:19:27 dist-test-slave-d3n8 krb5kdc[12851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:19:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:19:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:19:27.280040 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:37591
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:46009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:37591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:27.362223 12864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:27.362447 12864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:27.362473 12864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:27.363691 12864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:19:27.363740 12864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:27.363755 12864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:27.363767 12864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:19:27.363780 12864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:19:27.365372 12864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:46009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:37591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:37591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:27.365660 12864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:27.365917 12864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:27.371120 12869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:27.371161 12870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:27.371172 12872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:27.372157 12864 server_base.cc:1034] running on GCE node
I20260619 08:19:27.372488 12864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:27.372835 12864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:27.374017 12864 hybrid_clock.cc:648] HybridClock initialized: now 1781857167373996 us; error 43 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T08:19:27Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1307 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:19:27 dist-test-slave-d3n8 krb5kdc[12877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:19:27 dist-test-slave-d3n8 krb5kdc[12877](info): set up 2 sockets
Jun 19 08:19:27 dist-test-slave-d3n8 krb5kdc[12877](info): commencing operation
krb5kdc: starting...
W20260619 08:19:28.442094 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:19:28 dist-test-slave-d3n8 krb5kdc[12877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:19:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:19:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:19:28.574577 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:40109
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:40109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:28.655905 12890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:28.656172 12890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:28.656211 12890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:28.657413 12890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:19:28.657488 12890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:28.657512 12890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:28.657533 12890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:19:28.657564 12890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:19:28.659073 12890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:40109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:40109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:28.659373 12890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:28.659613 12890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:28.664825 12896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:28.664919 12895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:28.664825 12898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:28.665186 12890 server_base.cc:1034] running on GCE node
I20260619 08:19:28.665547 12890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:28.665900 12890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:28.667110 12890 hybrid_clock.cc:648] HybridClock initialized: now 1781857168667091 us; error 50 us; skew 500 ppm
Jun 19 08:19:28 dist-test-slave-d3n8 krb5kdc[12877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857168, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:28.675588 12890 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:19:28.676222 12890 webserver.cc:460] Webserver started at http://127.4.151.190:34079/ using document root <none> and password file <none>
I20260619 08:19:28.676487 12890 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:28.676558 12890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:28.676729 12890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:28.677942 12890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "4e1c8145799c44b19ece6fde66c53d25"
format_stamp: "Formatted at 2026-06-19 08:19:28 on dist-test-slave-d3n8"
server_key: "8efe9a3bb237734c8a8e34afba7a59fe"
server_key_iv: "c615b777f7abbc90c6de4b69e70389e1"
server_key_version: "encryptionkey@0"
I20260619 08:19:28.678333 12890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "4e1c8145799c44b19ece6fde66c53d25"
format_stamp: "Formatted at 2026-06-19 08:19:28 on dist-test-slave-d3n8"
server_key: "8efe9a3bb237734c8a8e34afba7a59fe"
server_key_iv: "c615b777f7abbc90c6de4b69e70389e1"
server_key_version: "encryptionkey@0"
I20260619 08:19:28.679768 12890 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:19:28.680720 12905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:28.680931 12890 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:28.681119 12890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "4e1c8145799c44b19ece6fde66c53d25"
format_stamp: "Formatted at 2026-06-19 08:19:28 on dist-test-slave-d3n8"
server_key: "8efe9a3bb237734c8a8e34afba7a59fe"
server_key_iv: "c615b777f7abbc90c6de4b69e70389e1"
server_key_version: "encryptionkey@0"
I20260619 08:19:28.681207 12890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:28.691219 12890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:28.693626 12890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:28.693823 12890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:28.698405 12890 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:40109
I20260619 08:19:28.698470 12957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:40109 every 8 connection(s)
I20260619 08:19:28.699038 12890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:19:28.699551 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 12890
I20260619 08:19:28.699668 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:19:28.699887 4702 external_mini_cluster.cc:1428] Setting key a4d4b011981d5966a0a41e85905073d4
I20260619 08:19:28.699965 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:28.702826 12958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25: Bootstrap starting.
I20260619 08:19:28.703462 12958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:28.703755 12958 log.cc:824] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:28.704439 12958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25: No bootstrap required, opened a new log
I20260619 08:19:28.706633 12958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1c8145799c44b19ece6fde66c53d25" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 40109 } }
I20260619 08:19:28.706821 12958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:28.706858 12958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e1c8145799c44b19ece6fde66c53d25, State: Initialized, Role: FOLLOWER
Jun 19 08:19:28 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:28.706981 12958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1c8145799c44b19ece6fde66c53d25" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 40109 } }
I20260619 08:19:28.707054 12958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:19:28.707091 12958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:19:28.707147 12958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:28.707891 12958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1c8145799c44b19ece6fde66c53d25" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 40109 } }
I20260619 08:19:28.708005 12958 leader_election.cc:302] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e1c8145799c44b19ece6fde66c53d25; no voters:
I20260619 08:19:28.708204 12958 leader_election.cc:288] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:19:28.708273 12963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:28.708482 12958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:19:28.708573 12963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [term 1 LEADER]: Becoming Leader. State: Replica: 4e1c8145799c44b19ece6fde66c53d25, State: Running, Role: LEADER
I20260619 08:19:28.708676 12963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1c8145799c44b19ece6fde66c53d25" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 40109 } }
I20260619 08:19:28.709097 12963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e1c8145799c44b19ece6fde66c53d25. Latest consensus state: current_term: 1 leader_uuid: "4e1c8145799c44b19ece6fde66c53d25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1c8145799c44b19ece6fde66c53d25" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 40109 } } }
I20260619 08:19:28.709334 12963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:28.709275 12964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e1c8145799c44b19ece6fde66c53d25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1c8145799c44b19ece6fde66c53d25" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 40109 } } }
I20260619 08:19:28.709784 12964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [sys.catalog]: This master's current role is: LEADER
W20260619 08:19:28.710793 12979 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:19:28.710857 12979 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:19:28.710927 12972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:19:28.711110 12972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:19:28.712105 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:28.700513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37454 (local address 127.4.151.190:40109)
0619 08:19:28.700723 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:28.700738 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:28.700768 (+ 30us) server_negotiation.cc:407] Connection header received
0619 08:19:28.701352 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:28.701360 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:28.701472 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:28.701624 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:28.702311 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:28.703308 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:28.704255 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:28.704730 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:28.704880 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:28.707600 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:28.707604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:28.707604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:28.707616 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:28.709828 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:28.710400 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:28.710403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:28.710403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:28.710488 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:28.710735 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:28.710736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:28.710736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:28.711161 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:28.711314 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:28.711763 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 08:19:28.711913 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":88,"threads_started":1}
I20260619 08:19:28.712554 12972 catalog_manager.cc:1347] Generated new cluster ID: b0113724d4a543acbe72508a72d118db
I20260619 08:19:28.712612 12972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:19:28.731990 12972 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:19:28.732770 12972 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:19:28.759569 12972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25: Generated new TSK 0
I20260619 08:19:28.759842 12972 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.151.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
I20260619 08:19:28.796911 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:40109
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:28.882745 12984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:28.882977 12984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:28.883008 12984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:28.884467 12984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:28.884519 12984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:28.884550 12984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:19:28.886314 12984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:40109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:28.886548 12984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:28.886780 12984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:28.892203 12989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:28.892295 12990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:28.892297 12992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:28.892611 12984 server_base.cc:1034] running on GCE node
I20260619 08:19:28.892835 12984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:28.893045 12984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:28.894220 12984 hybrid_clock.cc:648] HybridClock initialized: now 1781857168894208 us; error 35 us; skew 500 ppm
Jun 19 08:19:28 dist-test-slave-d3n8 krb5kdc[12877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857168, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:28.896727 12984 init.cc:377] Logged in from keytab as kudu/127.4.151.129@KRBTEST.COM (short username kudu)
I20260619 08:19:28.897238 12984 webserver.cc:460] Webserver started at http://127.4.151.129:33303/ using document root <none> and password file <none>
I20260619 08:19:28.897537 12984 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:28.897611 12984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:28.897773 12984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:28.898959 12984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "4ff470f878f94b54b593ed1a4f57c1ad"
format_stamp: "Formatted at 2026-06-19 08:19:28 on dist-test-slave-d3n8"
server_key: "d39ca60f640dbeb0e687ae6fa9b96765"
server_key_iv: "9ea03d7bfb35149966d76cd9b5fe47d4"
server_key_version: "encryptionkey@0"
I20260619 08:19:28.899349 12984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "4ff470f878f94b54b593ed1a4f57c1ad"
format_stamp: "Formatted at 2026-06-19 08:19:28 on dist-test-slave-d3n8"
server_key: "d39ca60f640dbeb0e687ae6fa9b96765"
server_key_iv: "9ea03d7bfb35149966d76cd9b5fe47d4"
server_key_version: "encryptionkey@0"
I20260619 08:19:28.900854 12984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:19:28.901959 12999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:28.902184 12984 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 08:19:28.902299 12984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "4ff470f878f94b54b593ed1a4f57c1ad"
format_stamp: "Formatted at 2026-06-19 08:19:28 on dist-test-slave-d3n8"
server_key: "d39ca60f640dbeb0e687ae6fa9b96765"
server_key_iv: "9ea03d7bfb35149966d76cd9b5fe47d4"
server_key_version: "encryptionkey@0"
I20260619 08:19:28.902385 12984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:28.913746 12984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:28.915982 12984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:28.916163 12984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:28.916416 12984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:28.916772 12984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:28.916828 12984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:28.916870 12984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:28.916929 12984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:28.922991 12984 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:37567
I20260619 08:19:28.923172 13112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:37567 every 8 connection(s)
I20260619 08:19:28.923411 12984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
Jun 19 08:19:28 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857168, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:28.932338 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 12984
I20260619 08:19:28.932492 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:19:28.932765 4702 external_mini_cluster.cc:1428] Setting key f9b68c254e27949accad844583934d4f
I20260619 08:19:28.933911 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:28.924290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:55311 (local address 127.4.151.190:40109)
0619 08:19:28.924339 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:28.924341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:28.924531 (+ 190us) server_negotiation.cc:407] Connection header received
0619 08:19:28.925091 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:28.925093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:28.925108 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:28.925173 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:28.926378 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:28.926922 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:28.928076 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:28.928309 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:28.928375 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:28.930775 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:28.930777 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:28.930777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:28.930784 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:28.932221 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:28.932699 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:28.932701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:28.932701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:28.932766 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:28.933051 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:28.933053 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:28.933053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:28.933198 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:28.933268 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:28.933657 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 08:19:28.933761 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:19:28.933935 13115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:28.924215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.129:55311)
0619 08:19:28.924388 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:28.924430 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:28.924925 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:28.925310 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:28.925314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:28.926038 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:28.926252 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:28.926258 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:28.927090 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:28.927091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:28.927927 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:28.927930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:28.928396 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:28.928397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:28.928552 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:28.929181 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:28.929189 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:28.930679 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:28.932313 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:28.932320 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:28.932322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:28.932621 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:28.932853 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:28.932854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:28.932854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:28.932971 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:28.933267 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:28.933274 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:28.933498 (+ 224us) client_negotiation.cc:769] Sending connection context
0619 08:19:28.933615 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 08:19:28.933754 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260619 08:19:28.934711 13113 heartbeater.cc:347] Connected to a master server at 127.4.151.190:40109
I20260619 08:19:28.934819 13113 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:28.935134 13113 heartbeater.cc:511] Master 127.4.151.190:40109 requested a full tablet report, sending...
I20260619 08:19:28.935671 12922 ts_manager.cc:194] Registered new tserver with Master: 4ff470f878f94b54b593ed1a4f57c1ad (127.4.151.129:37567)
I20260619 08:19:28.936627 12922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.129@KRBTEST.COM'} at 127.4.151.129:55311
WARNING: no policy specified for kudu/127.4.151.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
I20260619 08:19:28.960831 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:40109
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:29.045786 13118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:29.046030 13118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:29.046058 13118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:29.047407 13118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:29.047463 13118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:29.047497 13118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:19:29.049096 13118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:40109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:29.049436 13118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:29.049783 13118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:29.055248 13124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:29.055370 13126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:29.055315 13118 server_base.cc:1034] running on GCE node
W20260619 08:19:29.055238 13123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:29.055650 13118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:29.055964 13118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:29.057178 13118 hybrid_clock.cc:648] HybridClock initialized: now 1781857169057202 us; error 72 us; skew 500 ppm
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857169, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:29.060105 13118 init.cc:377] Logged in from keytab as kudu/127.4.151.130@KRBTEST.COM (short username kudu)
I20260619 08:19:29.060570 13118 webserver.cc:460] Webserver started at http://127.4.151.130:44039/ using document root <none> and password file <none>
I20260619 08:19:29.060830 13118 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:29.060910 13118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:29.061158 13118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:29.062350 13118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "26550ac830c14f3c93e3d5fdbca0c7c6"
format_stamp: "Formatted at 2026-06-19 08:19:29 on dist-test-slave-d3n8"
server_key: "249496c6b4ab13d67197a12b84106abc"
server_key_iv: "e2c0bcdac3a74eea55421f81b03db9fe"
server_key_version: "encryptionkey@0"
I20260619 08:19:29.062722 13118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "26550ac830c14f3c93e3d5fdbca0c7c6"
format_stamp: "Formatted at 2026-06-19 08:19:29 on dist-test-slave-d3n8"
server_key: "249496c6b4ab13d67197a12b84106abc"
server_key_iv: "e2c0bcdac3a74eea55421f81b03db9fe"
server_key_version: "encryptionkey@0"
I20260619 08:19:29.064116 13118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 08:19:29.064980 13133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:29.065202 13118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:29.065302 13118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "26550ac830c14f3c93e3d5fdbca0c7c6"
format_stamp: "Formatted at 2026-06-19 08:19:29 on dist-test-slave-d3n8"
server_key: "249496c6b4ab13d67197a12b84106abc"
server_key_iv: "e2c0bcdac3a74eea55421f81b03db9fe"
server_key_version: "encryptionkey@0"
I20260619 08:19:29.065411 13118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:29.079732 13118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:29.082367 13118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:29.082554 13118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:29.082806 13118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:29.083138 13118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:29.083194 13118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:29.083235 13118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:29.083261 13118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:29.089219 13118 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:40035
I20260619 08:19:29.089309 13246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:40035 every 8 connection(s)
I20260619 08:19:29.089655 13118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
I20260619 08:19:29.096045 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 13118
I20260619 08:19:29.096154 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:19:29.096369 4702 external_mini_cluster.cc:1428] Setting key 0ebebcec9e8139fc5bbd8b01ae3a4096
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857169, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:29.100558 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:29.090469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:44185 (local address 127.4.151.190:40109)
0619 08:19:29.090554 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:29.090556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:29.090726 (+ 170us) server_negotiation.cc:407] Connection header received
0619 08:19:29.091205 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:29.091206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:29.091216 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:29.091282 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:29.092357 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.092936 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.094077 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.094285 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.094354 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:29.097158 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:29.097160 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:29.097160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:29.097166 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:29.098966 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:29.099465 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:29.099467 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:29.099467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:29.099533 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:29.099814 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:29.099815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:29.099816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:29.099985 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:29.100058 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:29.100316 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 08:19:29.100416 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 08:19:29.100582 13249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:29.090440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.130:44185)
0619 08:19:29.090596 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:29.090638 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:29.091114 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:29.091354 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:29.091358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:29.091962 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:29.092166 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:29.092170 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.093050 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.093051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:29.093986 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:29.093989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.094372 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.094372 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:29.094505 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:29.095191 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:29.095200 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:29.097036 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:29.099135 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:29.099141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:29.099144 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:29.099382 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:29.099629 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:29.099630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:29.099630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:29.099743 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:29.100055 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:29.100060 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:29.100185 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 08:19:29.100272 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 08:19:29.100405 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260619 08:19:29.101279 13247 heartbeater.cc:347] Connected to a master server at 127.4.151.190:40109
I20260619 08:19:29.101423 13247 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:29.101608 13247 heartbeater.cc:511] Master 127.4.151.190:40109 requested a full tablet report, sending...
I20260619 08:19:29.102035 12922 ts_manager.cc:194] Registered new tserver with Master: 26550ac830c14f3c93e3d5fdbca0c7c6 (127.4.151.130:40035)
I20260619 08:19:29.102432 12922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.130@KRBTEST.COM'} at 127.4.151.130:44185
WARNING: no policy specified for kudu/127.4.151.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
I20260619 08:19:29.127444 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:40109
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:29.211781 13252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:29.212033 13252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:29.212076 13252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:29.213311 13252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:29.213385 13252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:29.213456 13252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:19:29.214977 13252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:40109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:29.215252 13252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:29.215492 13252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:29.220829 13260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:29.220881 13257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:29.220829 13258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:29.221097 13252 server_base.cc:1034] running on GCE node
I20260619 08:19:29.221304 13252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:29.221598 13252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:29.222855 13252 hybrid_clock.cc:648] HybridClock initialized: now 1781857169222832 us; error 46 us; skew 500 ppm
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857169, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:29.225780 13252 init.cc:377] Logged in from keytab as kudu/127.4.151.131@KRBTEST.COM (short username kudu)
I20260619 08:19:29.226248 13252 webserver.cc:460] Webserver started at http://127.4.151.131:34165/ using document root <none> and password file <none>
I20260619 08:19:29.226445 13252 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:29.226485 13252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:29.226583 13252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:29.227600 13252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "8dc0ea7b943e4c01a2da9c8c687bbce0"
format_stamp: "Formatted at 2026-06-19 08:19:29 on dist-test-slave-d3n8"
server_key: "cfdfbe0c11a8d8204fb24e3347fbaa2b"
server_key_iv: "202a0a4340aec82e0d72342096ba6490"
server_key_version: "encryptionkey@0"
I20260619 08:19:29.227895 13252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "8dc0ea7b943e4c01a2da9c8c687bbce0"
format_stamp: "Formatted at 2026-06-19 08:19:29 on dist-test-slave-d3n8"
server_key: "cfdfbe0c11a8d8204fb24e3347fbaa2b"
server_key_iv: "202a0a4340aec82e0d72342096ba6490"
server_key_version: "encryptionkey@0"
I20260619 08:19:29.229295 13252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 08:19:29.230264 13267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:29.230589 13252 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:29.230676 13252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "8dc0ea7b943e4c01a2da9c8c687bbce0"
format_stamp: "Formatted at 2026-06-19 08:19:29 on dist-test-slave-d3n8"
server_key: "cfdfbe0c11a8d8204fb24e3347fbaa2b"
server_key_iv: "202a0a4340aec82e0d72342096ba6490"
server_key_version: "encryptionkey@0"
I20260619 08:19:29.230728 13252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:29.251389 13252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:29.253861 13252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:29.253999 13252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:29.254204 13252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:29.254532 13252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:29.254573 13252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:29.254596 13252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:29.254649 13252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:29.260771 13252 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:41109
I20260619 08:19:29.260804 13380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:41109 every 8 connection(s)
I20260619 08:19:29.261210 13252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
I20260619 08:19:29.263006 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 13252
I20260619 08:19:29.263084 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:19:29.263264 4702 external_mini_cluster.cc:1428] Setting key e5f594263b82f20a659864196dd18001
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857169, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:29.272297 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:29.262126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:58357 (local address 127.4.151.190:40109)
0619 08:19:29.262209 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:29.262212 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:29.262286 (+ 74us) server_negotiation.cc:407] Connection header received
0619 08:19:29.262818 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:29.262819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:29.262829 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:29.262884 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:29.264713 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.265280 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.266393 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.266590 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.266650 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:29.268980 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:29.268982 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:29.268982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:29.268991 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:29.270521 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:29.271091 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:29.271093 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:29.271093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:29.271146 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:29.271454 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:29.271454 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:29.271455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:29.271659 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:29.271729 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:29.272156 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 08:19:29.272231 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 08:19:29.272455 13383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:29.262010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.131:58357)
0619 08:19:29.262171 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:29.262215 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:29.262722 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:29.263508 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:29.263512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:29.264382 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:29.264632 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:29.264637 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.265367 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.265368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:29.266318 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:29.266320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.266689 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.266690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:29.266798 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:29.267308 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:29.267316 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:29.268879 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:29.270616 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:29.270639 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:29.270642 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:29.270930 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:29.271235 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:29.271236 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:29.271237 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:29.271352 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:29.271760 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:29.271766 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:29.271895 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 08:19:29.272076 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 08:19:29.272216 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260619 08:19:29.273185 13381 heartbeater.cc:347] Connected to a master server at 127.4.151.190:40109
I20260619 08:19:29.273274 13381 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:29.273499 13381 heartbeater.cc:511] Master 127.4.151.190:40109 requested a full tablet report, sending...
I20260619 08:19:29.273833 12922 ts_manager.cc:194] Registered new tserver with Master: 8dc0ea7b943e4c01a2da9c8c687bbce0 (127.4.151.131:41109)
I20260619 08:19:29.274235 12922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.131@KRBTEST.COM'} at 127.4.151.131:58357
I20260619 08:19:29.275375 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.151.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.151.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.151.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.129.keytab.
Entry for principal oryx/127.4.151.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.129.keytab.
WARNING: no policy specified for oryx/127.4.151.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.151.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.151.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.130.keytab.
Entry for principal oryx/127.4.151.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.130.keytab.
WARNING: no policy specified for oryx/127.4.151.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.151.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.151.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.131.keytab.
Entry for principal oryx/127.4.151.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.131.keytab.
I20260619 08:19:29.362016 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 12984
I20260619 08:19:29.366694 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 13118
I20260619 08:19:29.371425 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 13252
I20260619 08:19:29.376341 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 12890
I20260619 08:19:29.382102 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:40109
--webserver_interface=127.4.151.190
--webserver_port=34079
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:40109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:29.467782 13390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:29.467998 13390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:29.468029 13390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:29.469914 13390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:19:29.469966 13390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:29.469981 13390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:29.469995 13390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:19:29.470013 13390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:19:29.471716 13390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:40109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:40109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=34079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:29.471982 13390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:29.472244 13390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:29.477535 13398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:29.477612 13396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:29.477545 13395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:29.477694 13390 server_base.cc:1034] running on GCE node
I20260619 08:19:29.478066 13390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:29.478412 13390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:29.479591 13390 hybrid_clock.cc:648] HybridClock initialized: now 1781857169479598 us; error 65 us; skew 500 ppm
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857169, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:29.482394 13390 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:19:29.482981 13390 webserver.cc:460] Webserver started at http://127.4.151.190:34079/ using document root <none> and password file <none>
I20260619 08:19:29.483259 13390 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:29.483353 13390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:29.485015 13390 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:29.485942 13405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:29.486192 13390 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 08:19:29.486312 13390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "4e1c8145799c44b19ece6fde66c53d25"
format_stamp: "Formatted at 2026-06-19 08:19:28 on dist-test-slave-d3n8"
server_key: "8efe9a3bb237734c8a8e34afba7a59fe"
server_key_iv: "c615b777f7abbc90c6de4b69e70389e1"
server_key_version: "encryptionkey@0"
I20260619 08:19:29.486680 13390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:29.507875 13390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:29.510105 13390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:29.510289 13390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:29.514628 13390 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:40109
I20260619 08:19:29.514681 13457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:40109 every 8 connection(s)
I20260619 08:19:29.515120 13390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:19:29.516686 13458 sys_catalog.cc:263] Verifying existing consensus state
I20260619 08:19:29.517401 13458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25: Bootstrap starting.
I20260619 08:19:29.517704 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 13390
I20260619 08:19:29.518092 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:37567
--local_ip_for_outbound_sockets=127.4.151.129
--tserver_master_addrs=127.4.151.190:40109
--webserver_port=33303
--webserver_interface=127.4.151.129
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.151.129
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:19:29.526290 13458 log.cc:824] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:29.528280 13458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:19:29.528594 13458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25: Bootstrap complete.
I20260619 08:19:29.530989 13458 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1c8145799c44b19ece6fde66c53d25" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 40109 } }
I20260619 08:19:29.531237 13458 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e1c8145799c44b19ece6fde66c53d25, State: Initialized, Role: FOLLOWER
I20260619 08:19:29.531415 13458 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1c8145799c44b19ece6fde66c53d25" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 40109 } }
I20260619 08:19:29.531522 13458 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:19:29.531587 13458 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:19:29.531639 13458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [term 1 FOLLOWER]: Advancing to term 2
I20260619 08:19:29.532759 13458 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1c8145799c44b19ece6fde66c53d25" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 40109 } }
I20260619 08:19:29.532889 13458 leader_election.cc:302] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e1c8145799c44b19ece6fde66c53d25; no voters:
I20260619 08:19:29.533105 13458 leader_election.cc:288] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 08:19:29.533160 13462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 08:19:29.533308 13462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [term 2 LEADER]: Becoming Leader. State: Replica: 4e1c8145799c44b19ece6fde66c53d25, State: Running, Role: LEADER
I20260619 08:19:29.533406 13458 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:19:29.533480 13462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1c8145799c44b19ece6fde66c53d25" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 40109 } }
I20260619 08:19:29.533809 13463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4e1c8145799c44b19ece6fde66c53d25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1c8145799c44b19ece6fde66c53d25" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 40109 } } }
I20260619 08:19:29.533917 13463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:29.534085 13464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e1c8145799c44b19ece6fde66c53d25. Latest consensus state: current_term: 2 leader_uuid: "4e1c8145799c44b19ece6fde66c53d25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1c8145799c44b19ece6fde66c53d25" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 40109 } } }
I20260619 08:19:29.534145 13464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:29.534313 13469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:19:29.534878 13469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:19:29.535049 13469 catalog_manager.cc:1259] Loaded cluster ID: b0113724d4a543acbe72508a72d118db
I20260619 08:19:29.535261 13469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:19:29.536538 13469 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:19:29.536798 13469 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4e1c8145799c44b19ece6fde66c53d25: Loaded TSK: 0
I20260619 08:19:29.537021 13469 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 08:19:29.607515 13460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:29.607755 13460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:29.607795 13460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:29.608981 13460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:29.609050 13460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:29.609097 13460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:19:29.610620 13460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.129.keytab
--principal=oryx/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:37567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=33303
--tserver_master_addrs=127.4.151.190:40109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:29.610885 13460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:29.611130 13460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:29.617165 13486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:29.617187 13488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:29.617187 13485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:29.617262 13460 server_base.cc:1034] running on GCE node
I20260619 08:19:29.617594 13460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:29.617895 13460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:29.619078 13460 hybrid_clock.cc:648] HybridClock initialized: now 1781857169619067 us; error 49 us; skew 500 ppm
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857169, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:29.622108 13460 init.cc:377] Logged in from keytab as oryx/127.4.151.129@KRBTEST.COM (short username oryx)
I20260619 08:19:29.622665 13460 webserver.cc:460] Webserver started at http://127.4.151.129:33303/ using document root <none> and password file <none>
I20260619 08:19:29.622975 13460 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:29.623059 13460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:29.624495 13460 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:29.625135 13495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:29.625355 13460 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:29.625474 13460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "4ff470f878f94b54b593ed1a4f57c1ad"
format_stamp: "Formatted at 2026-06-19 08:19:28 on dist-test-slave-d3n8"
server_key: "d39ca60f640dbeb0e687ae6fa9b96765"
server_key_iv: "9ea03d7bfb35149966d76cd9b5fe47d4"
server_key_version: "encryptionkey@0"
I20260619 08:19:29.625927 13460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:29.655793 13460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:29.658674 13460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:29.658843 13460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:29.659138 13460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:29.659521 13460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:29.659570 13460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:29.659603 13460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:29.659622 13460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:29.667201 13460 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:37567
I20260619 08:19:29.667302 13608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:37567 every 8 connection(s)
I20260619 08:19:29.667655 13460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
I20260619 08:19:29.673787 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 13460
I20260619 08:19:29.674111 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:40035
--local_ip_for_outbound_sockets=127.4.151.130
--tserver_master_addrs=127.4.151.190:40109
--webserver_port=44039
--webserver_interface=127.4.151.130
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.151.130
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:29.678507 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:29.668535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.129:39843)
0619 08:19:29.668705 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:29.668756 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:29.669257 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:29.669758 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:29.669762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:29.670492 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:29.670775 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:29.670782 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.672474 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.672475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:29.673515 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:29.673518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.674481 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.674482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:29.674663 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:29.675355 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:29.675366 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:29.678264 (+ 2898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
W20260619 08:19:29.678684 13612 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:29.668654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:39843 (local address 127.4.151.190:40109)
0619 08:19:29.668852 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:29.668861 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:29.668902 (+ 41us) server_negotiation.cc:407] Connection header received
0619 08:19:29.669515 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:29.669522 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:29.669579 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:29.669720 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:29.670913 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.672329 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.673628 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.674387 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.674502 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:29.678469 (+ 3967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.129:39843: BlockingRecv error: recv got EOF from 127.4.151.129:39843 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":103,"threads_started":1}
W20260619 08:19:29.678822 13609 heartbeater.cc:644] Failed to heartbeat to 127.4.151.190:40109 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.151.190:40109: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:29.684782 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:29.679075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.129:56315)
0619 08:19:29.679169 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:29.679179 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:29.679215 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:29.679401 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:29.679401 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:29.679555 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:29.679610 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:29.679612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.680610 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.680611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:29.681472 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:29.681474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.681983 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.681984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:29.682046 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:29.682596 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:29.682602 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:29.684713 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260619 08:19:29.684913 13612 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:29.679171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:56315 (local address 127.4.151.190:40109)
0619 08:19:29.679217 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:29.679219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:29.679227 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:19:29.679301 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:29.679302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:29.679317 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:29.679392 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:29.679690 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.680519 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.681559 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.681901 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.681981 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:29.684833 (+ 2852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.129:56315: BlockingRecv error: recv got EOF from 127.4.151.129:56315 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:29.690584 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:29.685127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.129:38175)
0619 08:19:29.685181 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:29.685193 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:29.685235 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:29.685416 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:29.685416 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:29.685552 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:29.685587 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:29.685589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.686547 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.686548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:29.687312 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:29.687314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.687974 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.687975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:29.688096 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:29.688566 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:29.688571 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:29.690510 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 08:19:29.690630 13612 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:29.685181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:38175 (local address 127.4.151.190:40109)
0619 08:19:29.685224 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:29.685226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:29.685235 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:19:29.685305 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:29.685306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:29.685318 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:29.685422 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:29.685669 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.686461 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.687398 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.687701 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.687767 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:29.690587 (+ 2820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.129:38175: BlockingRecv error: recv got EOF from 127.4.151.129:38175 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 08:19:29.690953 13609 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 08:19:29.761528 13613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:29.761775 13613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:29.761816 13613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:29.763055 13613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:29.763134 13613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:29.763180 13613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:19:29.764662 13613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.130.keytab
--principal=oryx/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:40035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=44039
--tserver_master_addrs=127.4.151.190:40109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:29.764951 13613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:29.765198 13613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:19:29.770747 13613 server_base.cc:1034] running on GCE node
W20260619 08:19:29.770715 13618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:29.770735 13619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:29.770728 13621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:29.771380 13613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:29.771664 13613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:29.772847 13613 hybrid_clock.cc:648] HybridClock initialized: now 1781857169772846 us; error 42 us; skew 500 ppm
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857169, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:29.775691 13613 init.cc:377] Logged in from keytab as oryx/127.4.151.130@KRBTEST.COM (short username oryx)
I20260619 08:19:29.776297 13613 webserver.cc:460] Webserver started at http://127.4.151.130:44039/ using document root <none> and password file <none>
I20260619 08:19:29.776542 13613 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:29.776639 13613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:29.778228 13613 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260619 08:19:29.778896 13628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:29.779165 13613 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:29.779276 13613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "26550ac830c14f3c93e3d5fdbca0c7c6"
format_stamp: "Formatted at 2026-06-19 08:19:29 on dist-test-slave-d3n8"
server_key: "249496c6b4ab13d67197a12b84106abc"
server_key_iv: "e2c0bcdac3a74eea55421f81b03db9fe"
server_key_version: "encryptionkey@0"
I20260619 08:19:29.779703 13613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:29.787616 13613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:29.790110 13613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:29.790287 13613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:29.790529 13613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:29.790853 13613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:29.790905 13613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:29.790961 13613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:29.790987 13613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:29.796852 13613 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:40035
I20260619 08:19:29.796900 13741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:40035 every 8 connection(s)
I20260619 08:19:29.797247 13613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
I20260619 08:19:29.799954 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 13613
I20260619 08:19:29.800335 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:41109
--local_ip_for_outbound_sockets=127.4.151.131
--tserver_master_addrs=127.4.151.190:40109
--webserver_port=34165
--webserver_interface=127.4.151.131
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.151.131
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
W20260619 08:19:29.805904 13612 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:29.798104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:38523 (local address 127.4.151.190:40109)
0619 08:19:29.798195 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:29.798198 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:29.798423 (+ 225us) server_negotiation.cc:407] Connection header received
0619 08:19:29.798916 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:29.798918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:29.798932 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:29.798986 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:29.799959 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.800745 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.802071 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.802283 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.802376 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:29.805847 (+ 3471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.130:38523: BlockingRecv error: recv got EOF from 127.4.151.130:38523 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 08:19:29.805968 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:29.798085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.130:38523)
0619 08:19:29.798306 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:29.798343 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:29.798828 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:29.799092 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:29.799096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:29.799666 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:29.799856 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:29.799861 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.800885 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.800886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:29.801886 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:29.801889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.802389 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.802390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:29.802521 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:29.803204 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:29.803212 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:29.805763 (+ 2551us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":116,"threads_started":1}
W20260619 08:19:29.806265 13742 heartbeater.cc:644] Failed to heartbeat to 127.4.151.190:40109 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.151.190:40109: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:29.811434 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:29.806520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.130:33339)
0619 08:19:29.806584 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:29.806595 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:29.806635 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:29.806784 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:29.806785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:29.806894 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:29.806936 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:29.806937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.807952 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.807953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:29.808442 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:29.808443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.808951 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.808951 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:29.809002 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:29.809543 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:29.809547 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:29.811385 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 08:19:29.811566 13612 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:29.806586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:33339 (local address 127.4.151.190:40109)
0619 08:19:29.806654 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:29.806656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:29.806665 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:19:29.806693 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:29.806694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:29.806706 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:29.806776 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:29.807008 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.807860 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.808528 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.808872 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.808951 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:29.811497 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.130:33339: BlockingRecv error: recv got EOF from 127.4.151.130:33339 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:29.816735 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:29.812113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.130:54857)
0619 08:19:29.812173 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:29.812185 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:29.812232 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:29.812380 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:29.812380 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:29.812552 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:29.812588 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:29.812589 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.813231 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.813231 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:29.813885 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:29.813887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.814274 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.814274 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:29.814328 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:29.814793 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:29.814798 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:29.816659 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 08:19:29.816843 13612 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:29.812190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:54857 (local address 127.4.151.190:40109)
0619 08:19:29.812251 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:29.812253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:29.812262 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:19:29.812296 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:29.812297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:29.812306 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:29.812376 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:29.812656 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.813132 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.813980 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.814170 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.814275 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:29.816749 (+ 2474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.130:54857: BlockingRecv error: recv got EOF from 127.4.151.130:54857 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 08:19:29.817025 13742 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 08:19:29.886617 13745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:29.886824 13745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:29.886849 13745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:29.888027 13745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:29.888070 13745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:29.888104 13745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:19:29.889801 13745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/krb5kdc/oryx_127.4.151.131.keytab
--principal=oryx/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:41109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=34165
--tserver_master_addrs=127.4.151.190:40109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:29.890053 13745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:29.890280 13745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:29.895697 13751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:29.895763 13750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:29.895695 13753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:29.895860 13745 server_base.cc:1034] running on GCE node
I20260619 08:19:29.896215 13745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:29.896457 13745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:29.897619 13745 hybrid_clock.cc:648] HybridClock initialized: now 1781857169897607 us; error 33 us; skew 500 ppm
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857169, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:29.903621 13745 init.cc:377] Logged in from keytab as oryx/127.4.151.131@KRBTEST.COM (short username oryx)
I20260619 08:19:29.904121 13745 webserver.cc:460] Webserver started at http://127.4.151.131:34165/ using document root <none> and password file <none>
I20260619 08:19:29.904331 13745 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:29.904385 13745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:29.905839 13745 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:29.906476 13760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:29.906673 13745 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:29.906745 13745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "8dc0ea7b943e4c01a2da9c8c687bbce0"
format_stamp: "Formatted at 2026-06-19 08:19:29 on dist-test-slave-d3n8"
server_key: "cfdfbe0c11a8d8204fb24e3347fbaa2b"
server_key_iv: "202a0a4340aec82e0d72342096ba6490"
server_key_version: "encryptionkey@0"
I20260619 08:19:29.907145 13745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:29.938665 13745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:29.940816 13745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:29.940934 13745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:29.941136 13745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:29.941505 13745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:29.941542 13745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:29.941566 13745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:29.941620 13745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:29.947424 13745 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:41109
I20260619 08:19:29.947499 13873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:41109 every 8 connection(s)
I20260619 08:19:29.947870 13745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:29.955757 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:29.948612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.131:34929)
0619 08:19:29.948791 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:29.948853 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:29.949328 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:29.949617 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:29.949620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:29.950176 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:29.950369 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:29.950373 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.951097 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.951110 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:29.951915 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:29.951918 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.952496 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.952497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:29.952609 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:29.953201 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:29.953210 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:29.955610 (+ 2400us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":116,"threads_started":1}
W20260619 08:19:29.955777 13612 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:29.948641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:34929 (local address 127.4.151.190:40109)
0619 08:19:29.948719 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:29.948721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:29.949034 (+ 313us) server_negotiation.cc:407] Connection header received
0619 08:19:29.949476 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:29.949477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:29.949489 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:29.949559 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:29.950484 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.950994 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.952199 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.952388 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.952452 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:29.955725 (+ 3273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.131:34929: BlockingRecv error: recv got EOF from 127.4.151.131:34929 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 08:19:29.956102 13874 heartbeater.cc:644] Failed to heartbeat to 127.4.151.190:40109 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.151.190:40109: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
I20260619 08:19:29.956537 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 13745
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:29.960976 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:29.956348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.131:45889)
0619 08:19:29.956418 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:29.956432 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:29.956485 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:29.956698 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:29.956699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:29.956816 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:29.956886 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:29.956888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.957742 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.957742 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:29.958249 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:29.958251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.958639 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.958640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:29.958709 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:29.959179 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:29.959184 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:29.960897 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 08:19:29.961345 13612 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:29.956533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:45889 (local address 127.4.151.190:40109)
0619 08:19:29.956584 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:29.956586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:29.956594 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:19:29.956620 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:29.956621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:29.956633 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:29.956691 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:29.956964 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.957656 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.958346 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.958553 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.958638 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:29.961265 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.131:45889: BlockingRecv error: recv got EOF from 127.4.151.131:45889 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 08:19:29.964356 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:29.956937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40300 (local address 127.4.151.190:40109)
0619 08:19:29.957041 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:29.957043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:29.957164 (+ 121us) server_negotiation.cc:407] Connection header received
0619 08:19:29.957193 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:29.957194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:29.957205 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:29.957270 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:29.957930 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.958488 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.959428 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.959673 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.959745 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:29.961014 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:29.961020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:29.961020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:29.961034 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:29.962821 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:29.963200 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:29.963204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:29.963204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:29.963312 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:29.963528 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:29.963529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:29.963529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:29.963831 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:29.964009 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:29.964081 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 08:19:29.964193 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":64,"threads_started":1}
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:29 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:29.965744 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:29.961349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.131:59483)
0619 08:19:29.961485 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:29.961494 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:29.961594 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:29.961720 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:29.961721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:29.961844 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:29.961869 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:29.961870 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.962540 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.962540 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:29.963021 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:29.963023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.963385 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.963386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:29.963441 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:29.963782 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:29.963786 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:29.965647 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121}
W20260619 08:19:29.965831 13612 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:29.961421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:59483 (local address 127.4.151.190:40109)
0619 08:19:29.961471 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:29.961473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:29.961582 (+ 109us) server_negotiation.cc:407] Connection header received
0619 08:19:29.961643 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:29.961644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:29.961653 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:29.961716 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:29.961964 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.962449 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.963117 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:29.963300 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:29.963381 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:29.965754 (+ 2373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.131:59483: BlockingRecv error: recv got EOF from 127.4.151.131:59483 (error 108)
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":33}
W20260619 08:19:29.966375 13874 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 08:19:30 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:30 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:30.696332 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:30.691416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.129:55231)
0619 08:19:30.691556 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:30.691568 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:30.691629 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:30.691778 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:30.691779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:30.691937 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:30.691987 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:30.691988 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:30.692694 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:30.692695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:30.693271 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:30.693274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:30.693692 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:30.693693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:30.693753 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:30.694231 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:30.694236 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:30.696262 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
W20260619 08:19:30.696473 13880 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:30.691488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:55231 (local address 127.4.151.190:40109)
0619 08:19:30.691631 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:30.691632 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:30.691672 (+ 40us) server_negotiation.cc:407] Connection header received
0619 08:19:30.691697 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:30.691697 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:30.691707 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:30.691760 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:30.692086 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:30.692619 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:30.693365 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:30.693603 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:30.693688 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:30.696361 (+ 2673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.129:55231: BlockingRecv error: recv got EOF from 127.4.151.129:55231 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
Jun 19 08:19:30 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:30 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:30.822299 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:30.817489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.130:51113)
0619 08:19:30.817635 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:30.817645 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:30.817707 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:30.817836 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:30.817836 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:30.818002 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:30.818053 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:30.818054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:30.818803 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:30.818804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:30.819367 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:30.819369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:30.819781 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:30.819782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:30.819825 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:30.820269 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:30.820273 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:30.822222 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
W20260619 08:19:30.822371 13880 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:30.817514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:51113 (local address 127.4.151.190:40109)
0619 08:19:30.817588 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:30.817591 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:30.817710 (+ 119us) server_negotiation.cc:407] Connection header received
0619 08:19:30.817755 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:30.817756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:30.817768 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:30.817818 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:30.818156 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:30.818710 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:30.819493 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:30.819698 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:30.819763 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:30.822323 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.130:51113: BlockingRecv error: recv got EOF from 127.4.151.130:51113 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 08:19:30 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:30 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:30.972234 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:30.966872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.131:48991)
0619 08:19:30.967101 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:30.967112 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:30.967180 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:30.967365 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:30.967366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:30.967530 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:30.967617 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:30.967619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:30.968442 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:30.968443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:30.969031 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:30.969034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:30.969563 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:30.969564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:30.969684 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:30.970218 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:30.970223 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:30.972160 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":102,"threads_started":1}
W20260619 08:19:30.972396 13880 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:30.967021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:48991 (local address 127.4.151.190:40109)
0619 08:19:30.967113 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:30.967115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:30.967184 (+ 69us) server_negotiation.cc:407] Connection header received
0619 08:19:30.967241 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:30.967242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:30.967255 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:30.967317 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:30.967728 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:30.968354 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:30.969153 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:30.969449 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:30.969559 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:30.972312 (+ 2753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.131:48991: BlockingRecv error: recv got EOF from 127.4.151.131:48991 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 08:19:31 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:31 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:31.701915 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:31.696865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.129:48879)
0619 08:19:31.697091 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:31.697105 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:31.697194 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:31.697334 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:31.697334 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:31.697555 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:31.697605 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:31.697607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:31.698341 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:31.698342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:31.699007 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:31.699011 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:31.699374 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:31.699375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:31.699433 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:31.699933 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:31.699938 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:31.701843 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":119,"threads_started":1}
W20260619 08:19:31.702020 13884 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:31.696952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:48879 (local address 127.4.151.190:40109)
0619 08:19:31.697147 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:31.697149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:31.697175 (+ 26us) server_negotiation.cc:407] Connection header received
0619 08:19:31.697243 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:31.697243 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:31.697254 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:31.697313 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:31.697705 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:31.698212 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:31.699099 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:31.699296 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:31.699352 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:31.701944 (+ 2592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.129:48879: BlockingRecv error: recv got EOF from 127.4.151.129:48879 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":120,"threads_started":1}
Jun 19 08:19:31 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:31 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:31.827755 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:31.822980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.130:35965)
0619 08:19:31.823116 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:31.823130 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:31.823218 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:31.823360 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:31.823361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:31.823522 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:31.823573 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:31.823575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:31.824320 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:31.824321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:31.824861 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:31.824862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:31.825235 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:31.825236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:31.825276 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:31.825840 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:31.825848 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:31.827694 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
W20260619 08:19:31.827939 13884 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:31.823041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:35965 (local address 127.4.151.190:40109)
0619 08:19:31.823148 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:31.823150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:31.823202 (+ 52us) server_negotiation.cc:407] Connection header received
0619 08:19:31.823272 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:31.823272 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:31.823283 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:31.823344 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:31.823669 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:31.824245 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:31.824955 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:31.825165 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:31.825227 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:31.827867 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.130:35965: BlockingRecv error: recv got EOF from 127.4.151.130:35965 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 19 08:19:31 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:31 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:31.978698 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:31.972873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.131:43501)
0619 08:19:31.973077 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:31.973093 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:31.973157 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:31.973317 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:31.973318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:31.973681 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:31.973738 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:31.973741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:31.974477 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:31.974478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:31.975007 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:31.975008 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:31.975385 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:31.975386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:31.975449 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:31.976251 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:31.976256 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:31.978595 (+ 2339us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":92,"threads_started":1}
W20260619 08:19:31.978754 13884 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:31.973117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:43501 (local address 127.4.151.190:40109)
0619 08:19:31.973171 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:31.973173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:31.973210 (+ 37us) server_negotiation.cc:407] Connection header received
0619 08:19:31.973236 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:31.973237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:31.973248 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:31.973291 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:31.973838 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:31.974372 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:31.975095 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:31.975292 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:31.975384 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:31.978686 (+ 3302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.131:43501: BlockingRecv error: recv got EOF from 127.4.151.131:43501 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 19 08:19:32 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:32 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:32.708052 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:32.702515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.129:43583)
0619 08:19:32.702773 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:32.702785 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:32.702840 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:32.703005 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:32.703006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:32.703193 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:32.703251 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:32.703253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:32.704230 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:32.704232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:32.704785 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:32.704787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:32.705194 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:32.705195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:32.705237 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:32.705762 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:32.705766 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:32.707989 (+ 2223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":183,"threads_started":1}
W20260619 08:19:32.708252 13888 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:32.702641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:43583 (local address 127.4.151.190:40109)
0619 08:19:32.702831 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:32.702833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:32.702843 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:19:32.702891 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:32.702891 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:32.702902 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:32.702983 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:32.703344 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:32.704138 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:32.704878 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:32.705112 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:32.705183 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:32.708176 (+ 2993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.129:43583: BlockingRecv error: recv got EOF from 127.4.151.129:43583 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":136,"threads_started":1}
Jun 19 08:19:32 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:32 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:32.833632 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:32.828378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.130:58693)
0619 08:19:32.828527 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:32.828539 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:32.828625 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:32.828772 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:32.828772 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:32.828934 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:32.828985 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:32.828986 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:32.829886 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:32.829888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:32.830505 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:32.830509 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:32.831059 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:32.831060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:32.831109 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:32.831637 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:32.831642 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:32.833572 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
W20260619 08:19:32.833806 13888 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:32.828450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:58693 (local address 127.4.151.190:40109)
0619 08:19:32.828555 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:32.828558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:32.828609 (+ 51us) server_negotiation.cc:407] Connection header received
0619 08:19:32.828699 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:32.828699 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:32.828710 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:32.828756 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:32.829082 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:32.829735 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:32.830597 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:32.830848 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:32.830913 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:32.833748 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.130:58693: BlockingRecv error: recv got EOF from 127.4.151.130:58693 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 08:19:32 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:32 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:32.984454 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:32.979289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.131:39999)
0619 08:19:32.979422 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:32.979434 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:32.979510 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:32.979690 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:32.979691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:32.979868 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:32.979918 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:32.979919 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:32.980667 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:32.980669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:32.981284 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:32.981286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:32.981693 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:32.981694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:32.981755 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:32.982297 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:32.982303 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:32.984355 (+ 2052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
W20260619 08:19:32.984535 13888 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:32.979411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:39999 (local address 127.4.151.190:40109)
0619 08:19:32.979501 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:32.979503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:32.979513 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:19:32.979589 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:32.979590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:32.979603 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:32.979679 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:32.980035 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:32.980579 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:32.981413 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:32.981614 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:32.981695 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:32.984439 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.131:39999: BlockingRecv error: recv got EOF from 127.4.151.131:39999 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 08:19:33 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:33 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:33.720561 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:33.708679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.129:57303)
0619 08:19:33.708837 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:33.708848 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:33.708924 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:33.709101 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:33.709103 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:33.709285 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:33.709335 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:33.709336 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:33.710177 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:33.710178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:33.710708 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:33.710710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:33.711085 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:33.711086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:33.711142 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:33.718651 (+ 7509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:33.718656 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:33.720498 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":101,"threads_started":1}
W20260619 08:19:33.720734 13892 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:33.708792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:57303 (local address 127.4.151.190:40109)
0619 08:19:33.708941 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:33.708943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:33.708951 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:19:33.708975 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:33.708975 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:33.708984 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:33.709042 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:33.709509 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:33.710099 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:33.710785 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:33.710986 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:33.711049 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:33.720638 (+ 9589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.129:57303: BlockingRecv error: recv got EOF from 127.4.151.129:57303 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":74,"threads_started":1}
Jun 19 08:19:33 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:33 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:33.839543 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:33.834313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.130:47861)
0619 08:19:33.834474 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:33.834485 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:33.834584 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:33.834734 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:33.834735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:33.834947 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:33.835014 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:33.835017 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:33.835683 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:33.835685 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:33.836282 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:33.836284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:33.836699 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:33.836700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:33.836750 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:33.837233 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:33.837237 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:33.839446 (+ 2209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
W20260619 08:19:33.839627 13892 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:33.834388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:47861 (local address 127.4.151.190:40109)
0619 08:19:33.834459 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:33.834462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:33.834573 (+ 111us) server_negotiation.cc:407] Connection header received
0619 08:19:33.834637 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:33.834638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:33.834648 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:33.834716 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:33.835099 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:33.835606 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:33.836408 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:33.836604 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:33.836679 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:33.839562 (+ 2883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.130:47861: BlockingRecv error: recv got EOF from 127.4.151.130:47861 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 08:19:33 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:33 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:33.990263 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:33.984975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.131:33901)
0619 08:19:33.985192 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:33.985208 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:33.985284 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:33.985664 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:33.985665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:33.985848 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:33.985971 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:33.985974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:33.986675 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:33.986676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:33.987205 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:33.987207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:33.987649 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:33.987650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:33.987711 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:33.988182 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:33.988187 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:33.990163 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":106,"threads_started":1}
W20260619 08:19:33.990322 13892 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:33.985076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:33901 (local address 127.4.151.190:40109)
0619 08:19:33.985181 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:33.985183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:33.985268 (+ 85us) server_negotiation.cc:407] Connection header received
0619 08:19:33.985342 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:33.985343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:33.985356 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:33.985650 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:33.986072 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:33.986591 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:33.987303 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:33.987569 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:33.987651 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:33.990241 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.131:33901: BlockingRecv error: recv got EOF from 127.4.151.131:33901 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 08:19:34 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:34 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:34.725997 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:34.721075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.129:43601)
0619 08:19:34.721261 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:34.721272 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:34.721325 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:34.721498 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:34.721499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:34.721666 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:34.721717 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:34.721718 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:34.722535 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:34.722536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:34.723134 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:34.723136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:34.723564 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:34.723565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:34.723620 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:34.724085 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:34.724090 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:34.725935 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":77,"threads_started":1}
W20260619 08:19:34.726135 13896 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:34.721185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:43601 (local address 127.4.151.190:40109)
0619 08:19:34.721342 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:34.721344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:34.721353 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:19:34.721401 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:34.721402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:34.721413 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:34.721484 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:34.721797 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:34.722432 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:34.723265 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:34.723490 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:34.723564 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:34.726042 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.129:43601: BlockingRecv error: recv got EOF from 127.4.151.129:43601 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
Jun 19 08:19:34 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:34 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:34.844858 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:34.840011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.130:44887)
0619 08:19:34.840182 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:34.840196 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:34.840270 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:34.840396 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:34.840397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:34.840586 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:34.840635 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:34.840637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:34.841340 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:34.841341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:34.841908 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:34.841910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:34.842322 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:34.842323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:34.842393 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:34.842910 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:34.842915 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:34.844759 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":120,"threads_started":1}
W20260619 08:19:34.844910 13896 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:34.840154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:44887 (local address 127.4.151.190:40109)
0619 08:19:34.840206 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:34.840208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:34.840252 (+ 44us) server_negotiation.cc:407] Connection header received
0619 08:19:34.840328 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:34.840329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:34.840341 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:34.840392 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:34.840729 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:34.841243 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:34.842023 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:34.842237 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:34.842318 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:34.844847 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.130:44887: BlockingRecv error: recv got EOF from 127.4.151.130:44887 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 19 08:19:34 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:34 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:34.996121 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:34.990788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.131:48947)
0619 08:19:34.990940 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:34.990958 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:34.991013 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:34.991214 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:34.991214 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:34.991382 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:34.991431 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:34.991433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:34.992184 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:34.992185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:34.992722 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:34.992724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:34.993131 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:34.993132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:34.993185 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:34.993673 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:34.993677 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:34.996020 (+ 2343us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
W20260619 08:19:34.996230 13896 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:34.990883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:48947 (local address 127.4.151.190:40109)
0619 08:19:34.990951 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:34.990953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:34.991003 (+ 50us) server_negotiation.cc:407] Connection header received
0619 08:19:34.991121 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:34.991122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:34.991134 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:34.991193 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:34.991533 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:34.992035 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:34.992812 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:34.993007 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:34.993124 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:34.996131 (+ 3007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.131:48947: BlockingRecv error: recv got EOF from 127.4.151.131:48947 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 08:19:35 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:35 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:35.731699 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:35.726584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.129:35261)
0619 08:19:35.726739 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:35.726749 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:35.726800 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:35.726959 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:35.726960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:35.727119 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:35.727168 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:35.727169 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:35.727833 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:35.727834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:35.728423 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:35.728426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:35.728813 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:35.728814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:35.728862 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:35.729343 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:35.729347 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:35.731637 (+ 2290us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
W20260619 08:19:35.731889 13900 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:35.726680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:35261 (local address 127.4.151.190:40109)
0619 08:19:35.726810 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:35.726812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:35.726821 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:19:35.726854 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:35.726855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:35.726865 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:35.726929 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:35.727244 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:35.727753 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:35.728518 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:35.728724 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:35.728782 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:35.731797 (+ 3015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.129:35261: BlockingRecv error: recv got EOF from 127.4.151.129:35261 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
Jun 19 08:19:35 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:35 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:35.850257 13901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:35.845470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.130:38317)
0619 08:19:35.845660 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:35.845674 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:35.845783 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:35.845960 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:35.845960 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:35.846147 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:35.846194 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:35.846196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:35.846913 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:35.846914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:35.847463 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:35.847466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:35.847855 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:35.847856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:35.847912 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:35.848366 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:35.848370 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:35.850157 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":91,"threads_started":1}
W20260619 08:19:35.850374 13900 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:35.845574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:38317 (local address 127.4.151.190:40109)
0619 08:19:35.845664 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:35.845666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:35.845761 (+ 95us) server_negotiation.cc:407] Connection header received
0619 08:19:35.845866 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:35.845867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:35.845881 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:35.845955 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:35.846287 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:35.846822 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:35.847567 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:35.847766 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:35.847854 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:35.850310 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.130:38317: BlockingRecv error: recv got EOF from 127.4.151.130:38317 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 08:19:35 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:35 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:36.002394 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:35.996668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.131:34651)
0619 08:19:35.996826 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:35.996863 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:35.996924 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:35.997217 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:35.997218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:35.997585 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:35.997648 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:35.997650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:35.998402 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:35.998403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:35.999046 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:35.999048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:35.999487 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:35.999488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:35.999559 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:36.000112 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:36.000119 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:36.002288 (+ 2169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":98,"threads_started":1}
W20260619 08:19:36.002435 13900 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:35.996793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:34651 (local address 127.4.151.190:40109)
0619 08:19:35.996868 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:35.996870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:35.996914 (+ 44us) server_negotiation.cc:407] Connection header received
0619 08:19:35.997103 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:35.997104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:35.997118 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:35.997185 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:35.997752 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:35.998309 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:35.999150 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:35.999378 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:35.999487 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:36.002372 (+ 2885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.131:34651: BlockingRecv error: recv got EOF from 127.4.151.131:34651 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 19 08:19:36 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:36 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:36.737169 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:36.732168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.129:45811)
0619 08:19:36.732413 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:36.732428 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:36.732482 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:36.732628 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:36.732628 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:36.732787 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:36.732837 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:36.732838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:36.733687 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:36.733688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:36.734239 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:36.734240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:36.734651 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:36.734652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:36.734718 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:36.735201 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:36.735206 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:36.737099 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":190,"threads_started":1}
W20260619 08:19:36.737357 13904 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:36.732267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:45811 (local address 127.4.151.190:40109)
0619 08:19:36.732413 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:36.732415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:36.732471 (+ 56us) server_negotiation.cc:407] Connection header received
0619 08:19:36.732539 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:36.732540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:36.732550 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:36.732605 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:36.732905 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:36.733597 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:36.734315 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:36.734558 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:36.734628 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:36.737240 (+ 2612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.129:45811: BlockingRecv error: recv got EOF from 127.4.151.129:45811 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
Jun 19 08:19:36 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:36 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:36.855715 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:36.850881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.130:48867)
0619 08:19:36.851071 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:36.851085 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:36.851196 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:36.851316 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:36.851317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:36.851498 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:36.851553 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:36.851555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:36.852266 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:36.852267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:36.852793 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:36.852794 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:36.853167 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:36.853168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:36.853232 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:36.853747 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:36.853752 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:36.855613 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":83,"threads_started":1}
W20260619 08:19:36.855798 13904 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:36.850936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:48867 (local address 127.4.151.190:40109)
0619 08:19:36.850987 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:36.850990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:36.851207 (+ 217us) server_negotiation.cc:407] Connection header received
0619 08:19:36.851245 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:36.851246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:36.851259 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:36.851308 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:36.851671 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:36.852178 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:36.852880 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:36.853087 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:36.853169 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:36.855724 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.130:48867: BlockingRecv error: recv got EOF from 127.4.151.130:48867 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Jun 19 08:19:37 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:37 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:37.008378 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:37.003077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.131:33385)
0619 08:19:37.003244 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:37.003263 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:37.003322 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:37.003557 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:37.003559 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:37.003729 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:37.003783 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:37.003785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:37.004487 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:37.004488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:37.005085 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:37.005087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:37.005449 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:37.005450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:37.005501 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:37.006061 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:37.006066 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:37.008308 (+ 2242us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":109,"threads_started":1}
W20260619 08:19:37.008524 13904 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:37.003188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:33385 (local address 127.4.151.190:40109)
0619 08:19:37.003281 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:37.003284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:37.003300 (+ 16us) server_negotiation.cc:407] Connection header received
0619 08:19:37.003417 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:37.003419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:37.003432 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:37.003498 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:37.003876 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:37.004395 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:37.005180 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:37.005392 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:37.005448 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:37.008430 (+ 2982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.131:33385: BlockingRecv error: recv got EOF from 127.4.151.131:33385 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 08:19:37 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:37 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:37.742808 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:37.737717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.129:37633)
0619 08:19:37.737883 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:37.737894 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:37.737946 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:37.738133 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:37.738134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:37.738299 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:37.738350 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:37.738352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:37.739093 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:37.739094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:37.739621 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:37.739622 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:37.740085 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:37.740086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:37.740131 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:37.740683 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:37.740688 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:37.742746 (+ 2058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":107,"threads_started":1}
W20260619 08:19:37.743054 13908 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:37.737860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:37633 (local address 127.4.151.190:40109)
0619 08:19:37.737995 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:37.737997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:37.738006 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:19:37.738038 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:37.738038 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:37.738048 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:37.738113 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:37.738437 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:37.739010 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:37.739703 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:37.739978 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:37.740037 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:37.742982 (+ 2945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.129:37633: BlockingRecv error: recv got EOF from 127.4.151.129:37633 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
Jun 19 08:19:37 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:37 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:37.867167 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:37.856278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.130:32967)
0619 08:19:37.856427 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:37.856441 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:37.856524 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:37.856959 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:37.856961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:37.857223 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:37.857295 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:37.857297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:37.859078 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:37.859079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:37.860092 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:37.860096 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:37.860679 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:37.860680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:37.860758 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:37.861362 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:37.861385 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:37.867064 (+ 5679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
W20260619 08:19:37.867311 13908 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:37.856335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:32967 (local address 127.4.151.190:40109)
0619 08:19:37.856409 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:37.856412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:37.856523 (+ 111us) server_negotiation.cc:407] Connection header received
0619 08:19:37.856843 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:37.856845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:37.856865 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:37.856944 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:37.857401 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:37.858974 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:37.860205 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:37.860574 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:37.860673 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:37.867236 (+ 6563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.130:32967: BlockingRecv error: recv got EOF from 127.4.151.130:32967 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 08:19:38 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:38 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:38.014508 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:38.009077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.131:40247)
0619 08:19:38.009298 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:38.009316 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:38.009366 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:38.009596 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:38.009597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:38.009762 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:38.009811 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:38.009812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:38.010562 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:38.010563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:38.011172 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:38.011174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:38.011562 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:38.011563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:38.011620 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:38.012137 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:38.012142 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:38.014408 (+ 2266us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":58,"threads_started":1}
W20260619 08:19:38.014564 13908 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:38.009329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:40247 (local address 127.4.151.190:40109)
0619 08:19:38.009423 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:38.009425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:38.009434 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:19:38.009461 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:38.009461 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:38.009473 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:38.009536 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:38.009916 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:38.010490 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:38.011265 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:38.011470 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:38.011541 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:38.014498 (+ 2957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.131:40247: BlockingRecv error: recv got EOF from 127.4.151.131:40247 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 08:19:38 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:38 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:38.753031 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:38.743518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.129:43885)
0619 08:19:38.743713 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:38.743730 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:38.743806 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:38.744048 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:38.744049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:38.744221 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:38.744273 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:38.744274 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:38.745473 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:38.745474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:38.746015 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:38.746017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:38.746409 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:38.746410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:38.746449 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:38.750648 (+ 4199us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:38.750654 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:38.752924 (+ 2270us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":83,"threads_started":1}
W20260619 08:19:38.753216 13912 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:38.743683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:43885 (local address 127.4.151.190:40109)
0619 08:19:38.743863 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:38.743865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:38.743876 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:19:38.743905 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:38.743905 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:38.743917 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:38.743994 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:38.744489 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:38.745338 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:38.746119 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:38.746332 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:38.746401 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:38.753143 (+ 6742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.129:43885: BlockingRecv error: recv got EOF from 127.4.151.129:43885 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":75,"threads_started":1}
Jun 19 08:19:38 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:38 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:38.872851 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:38.867665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.130:57127)
0619 08:19:38.867814 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:38.867832 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:38.867888 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:38.868095 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:38.868095 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:38.868257 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:38.868308 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:38.868309 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:38.869026 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:38.869026 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:38.869647 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:38.869650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:38.870067 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:38.870068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:38.870119 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:38.870658 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:38.870662 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:38.872769 (+ 2107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
W20260619 08:19:38.872926 13912 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:38.867763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:57127 (local address 127.4.151.190:40109)
0619 08:19:38.867828 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:38.867831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:38.867878 (+ 47us) server_negotiation.cc:407] Connection header received
0619 08:19:38.867935 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:38.867936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:38.867951 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:38.868013 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:38.868410 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:38.868917 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:38.869753 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:38.869950 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:38.870015 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:38.872878 (+ 2863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.130:57127: BlockingRecv error: recv got EOF from 127.4.151.130:57127 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 08:19:39 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:39 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:39.020385 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:39.015147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.131:33899)
0619 08:19:39.015291 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:39.015302 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:39.015365 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:39.015590 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:39.015591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:39.015774 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:39.015822 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:39.015824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:39.016750 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:39.016752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:39.017422 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:39.017425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:39.017870 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:39.017871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:39.017927 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:39.018444 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:39.018450 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:39.020286 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
W20260619 08:19:39.020509 13912 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:39.015354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:33899 (local address 127.4.151.190:40109)
0619 08:19:39.015429 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:39.015431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:39.015443 (+ 12us) server_negotiation.cc:407] Connection header received
0619 08:19:39.015470 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:39.015470 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:39.015494 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:39.015569 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:39.015919 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:39.016585 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:39.017581 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:39.017782 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:39.017857 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:39.020434 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.131:33899: BlockingRecv error: recv got EOF from 127.4.151.131:33899 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 08:19:39 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:39 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:39.758466 13915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:39.753733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.129:47337)
0619 08:19:39.753900 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:39.753912 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:39.753979 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:39.754179 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:39.754181 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:39.754405 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:39.754465 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:39.754467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:39.755180 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:39.755181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:39.755718 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:39.755720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:39.756104 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:39.756105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:39.756148 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:39.756620 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:39.756624 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:39.758404 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":98,"threads_started":1}
W20260619 08:19:39.758651 13916 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:39.753822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:47337 (local address 127.4.151.190:40109)
0619 08:19:39.753932 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:39.753934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:39.753982 (+ 48us) server_negotiation.cc:407] Connection header received
0619 08:19:39.754047 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:39.754048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:39.754061 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:39.754153 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:39.754545 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:39.755108 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:39.755812 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:39.756013 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:39.756088 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:39.758571 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.129:47337: BlockingRecv error: recv got EOF from 127.4.151.129:47337 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":43,"threads_started":1}
Jun 19 08:19:39 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:39 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:39.879186 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:39.873684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.130:51171)
0619 08:19:39.873831 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:39.873868 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:39.873959 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:39.874243 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:39.874244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:39.874433 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:39.874485 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:39.874487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:39.875254 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:39.875256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:39.875973 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:39.875976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:39.876441 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:39.876442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:39.876514 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:39.877098 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:39.877105 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:39.879077 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
W20260619 08:19:39.879243 13916 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:39.873830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:51171 (local address 127.4.151.190:40109)
0619 08:19:39.873920 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:39.873923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:39.873941 (+ 18us) server_negotiation.cc:407] Connection header received
0619 08:19:39.874130 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:39.874131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:39.874144 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:39.874223 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:39.874580 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:39.875153 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:39.876121 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:39.876347 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:39.876432 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:39.879172 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.130:51171: BlockingRecv error: recv got EOF from 127.4.151.130:51171 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 08:19:40 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:40 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:40.026191 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:40.021022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.131:60145)
0619 08:19:40.021230 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:40.021246 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:40.021313 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:40.021494 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:40.021495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:40.021790 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:40.021838 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:40.021840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:40.022512 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:40.022513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:40.023038 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:40.023039 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:40.023396 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:40.023397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:40.023439 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:40.023899 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:40.023903 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:40.026091 (+ 2188us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":96,"threads_started":1}
W20260619 08:19:40.026237 13916 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:40.021206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:60145 (local address 127.4.151.190:40109)
0619 08:19:40.021277 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:40.021279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:40.021294 (+ 15us) server_negotiation.cc:407] Connection header received
0619 08:19:40.021415 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:40.021416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:40.021430 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:40.021491 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:40.021912 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:40.022425 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:40.023129 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:40.023324 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:40.023382 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:40.026172 (+ 2790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.131:60145: BlockingRecv error: recv got EOF from 127.4.151.131:60145 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 08:19:40 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:40 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:40.763895 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:40.759006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.129:42827)
0619 08:19:40.759218 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:40.759231 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:40.759307 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:40.759500 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:40.759502 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:40.759681 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:40.759726 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:40.759728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:40.760435 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:40.760436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:40.760988 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:40.760990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:40.761481 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:40.761482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:40.761530 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:40.761967 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:40.761971 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:40.763826 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":118,"threads_started":1}
W20260619 08:19:40.763981 13920 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:40.759095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:42827 (local address 127.4.151.190:40109)
0619 08:19:40.759224 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:40.759226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:40.759304 (+ 78us) server_negotiation.cc:407] Connection header received
0619 08:19:40.759367 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:40.759368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:40.759380 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:40.759440 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:40.759807 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:40.760347 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:40.761137 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:40.761333 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:40.761411 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:40.763927 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.129:42827: BlockingRecv error: recv got EOF from 127.4.151.129:42827 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":52,"threads_started":1}
Jun 19 08:19:40 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:40 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:40.885195 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:40.879747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.130:40089)
0619 08:19:40.879928 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:40.879941 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:40.880103 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:40.880242 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:40.880243 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:40.880437 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:40.880494 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:40.880496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:40.881179 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:40.881179 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:40.881824 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:40.881827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:40.882310 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:40.882311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:40.882364 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:40.882925 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:40.883049 (+ 124us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:40.885120 (+ 2071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
W20260619 08:19:40.885437 13920 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:40.879867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:40089 (local address 127.4.151.190:40109)
0619 08:19:40.879960 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:40.879962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:40.880082 (+ 120us) server_negotiation.cc:407] Connection header received
0619 08:19:40.880164 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:40.880165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:40.880179 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:40.880226 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:40.880580 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:40.881095 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:40.881979 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:40.882193 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:40.882255 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:40.885331 (+ 3076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.130:40089: BlockingRecv error: recv got EOF from 127.4.151.130:40089 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 08:19:41 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:41 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:41.031858 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:41.026711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.131:46805)
0619 08:19:41.026852 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:41.026867 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:41.027014 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:41.027196 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:41.027197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:41.027378 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:41.027428 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:41.027429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:41.028096 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:41.028097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:41.028654 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:41.028656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:41.029033 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:41.029034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:41.029097 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:41.029684 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:41.029689 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:41.031781 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
W20260619 08:19:41.032176 13920 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:41.026834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:46805 (local address 127.4.151.190:40109)
0619 08:19:41.026900 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:41.026902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:41.026912 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:19:41.027106 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:41.027112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:41.027125 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:41.027189 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:41.027510 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:41.028009 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:41.028744 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:41.028947 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:41.029032 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:41.031906 (+ 2874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.131:46805: BlockingRecv error: recv got EOF from 127.4.151.131:46805 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 19 08:19:41 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:41 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:41.770200 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:41.764544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.129:55247)
0619 08:19:41.764698 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:41.764710 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:41.764773 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:41.764978 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:41.764979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:41.765227 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:41.765288 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:41.765290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:41.766278 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:41.766279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:41.766816 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:41.766818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:41.767290 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:41.767291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:41.767351 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:41.767920 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:41.767925 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:41.770136 (+ 2211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
W20260619 08:19:41.770452 13924 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:41.764670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:55247 (local address 127.4.151.190:40109)
0619 08:19:41.764811 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:41.764813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:41.764822 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:19:41.764849 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:41.764849 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:41.764859 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:41.764934 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:41.765443 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:41.766192 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:41.766940 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:41.767191 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:41.767255 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:41.770359 (+ 3104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.129:55247: BlockingRecv error: recv got EOF from 127.4.151.129:55247 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
Jun 19 08:19:41 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:41 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:41.891232 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:41.885799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.130:55189)
0619 08:19:41.885950 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:41.885980 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:41.886031 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:41.886273 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:41.886273 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:41.886502 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:41.886565 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:41.886568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:41.887316 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:41.887317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:41.887873 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:41.887875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:41.888282 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:41.888283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:41.888332 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:41.888886 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:41.888892 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:41.891116 (+ 2224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":96,"threads_started":1}
W20260619 08:19:41.891291 13924 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:41.885907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:55189 (local address 127.4.151.190:40109)
0619 08:19:41.886015 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:41.886017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:41.886027 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:19:41.886150 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:41.886152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:41.886165 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:41.886231 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:41.886673 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:41.887226 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:41.887971 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:41.888194 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:41.888264 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:41.891219 (+ 2955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.130:55189: BlockingRecv error: recv got EOF from 127.4.151.130:55189 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 19 08:19:42 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:42 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:42.037812 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:42.032718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.131:49155)
0619 08:19:42.032876 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:42.032888 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:42.032958 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:42.033146 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:42.033147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:42.033344 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:42.033439 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:42.033441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:42.034203 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:42.034204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:42.034878 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:42.034880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:42.035231 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:42.035232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:42.035272 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:42.035708 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:42.035714 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:42.037752 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
W20260619 08:19:42.037957 13924 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:42.032817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:49155 (local address 127.4.151.190:40109)
0619 08:19:42.032869 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:42.032871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:42.032967 (+ 96us) server_negotiation.cc:407] Connection header received
0619 08:19:42.033003 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:42.033004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:42.033030 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:42.033094 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:42.033560 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:42.034074 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:42.034955 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:42.035164 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:42.035223 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:42.037863 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.131:49155: BlockingRecv error: recv got EOF from 127.4.151.131:49155 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Jun 19 08:19:42 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:42 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:42.775666 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:42.770722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.129:48545)
0619 08:19:42.770866 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:42.770877 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:42.770931 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:42.771216 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:42.771216 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:42.771387 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:42.771447 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:42.771448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:42.772143 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:42.772144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:42.772685 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:42.772686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:42.773022 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:42.773023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:42.773062 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:42.773669 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:42.773674 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:42.775598 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
W20260619 08:19:42.775892 13928 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:42.770859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:48545 (local address 127.4.151.190:40109)
0619 08:19:42.771063 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:42.771065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:42.771075 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:19:42.771100 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:42.771101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:42.771112 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:42.771190 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:42.771551 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:42.772052 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:42.772759 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:42.772958 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:42.773009 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:42.775807 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.129:48545: BlockingRecv error: recv got EOF from 127.4.151.129:48545 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
Jun 19 08:19:42 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:42 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:42.897677 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:42.891809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.130:46017)
0619 08:19:42.891960 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:42.891974 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:42.892047 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:42.892422 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:42.892424 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:42.892591 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:42.892641 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:42.892643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:42.893612 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:42.893612 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:42.894175 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:42.894177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:42.894561 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:42.894562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:42.894604 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:42.895217 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:42.895222 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:42.897614 (+ 2392us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":57,"threads_started":1}
W20260619 08:19:42.897765 13928 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:42.892077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:46017 (local address 127.4.151.190:40109)
0619 08:19:42.892176 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:42.892178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:42.892188 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:19:42.892235 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:42.892236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:42.892256 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:42.892314 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:42.892784 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:42.893357 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:42.894268 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:42.894490 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:42.894549 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:42.897716 (+ 3167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.130:46017: BlockingRecv error: recv got EOF from 127.4.151.130:46017 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 08:19:43 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:43 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:43.043139 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:43.038376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.131:38827)
0619 08:19:43.038519 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:43.038531 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:43.038595 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:43.038714 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:43.038715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:43.038875 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:43.038923 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:43.038924 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:43.039584 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:43.039585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:43.040242 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:43.040245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:43.040555 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:43.040555 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:43.040594 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:43.041074 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:43.041079 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:43.043004 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
W20260619 08:19:43.043180 13928 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:43.038423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:38827 (local address 127.4.151.190:40109)
0619 08:19:43.038486 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:43.038489 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:43.038599 (+ 110us) server_negotiation.cc:407] Connection header received
0619 08:19:43.038646 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:43.038647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:43.038659 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:43.038699 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:43.038978 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:43.039507 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:43.040314 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:43.040491 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:43.040542 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:43.043131 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.131:38827: BlockingRecv error: recv got EOF from 127.4.151.131:38827 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 08:19:43 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:43 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:43.784548 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:43.776174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.129:46101)
0619 08:19:43.776345 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:43.776361 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:43.776415 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:43.776618 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:43.776618 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:43.776794 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:43.776844 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:43.776846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:43.777650 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:43.777651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:43.778197 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:43.778199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:43.778605 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:43.778606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:43.778659 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:43.782650 (+ 3991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:43.782656 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:43.784487 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":109,"threads_started":1}
W20260619 08:19:43.784840 13932 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:43.776306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:46101 (local address 127.4.151.190:40109)
0619 08:19:43.776446 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:43.776449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:43.776465 (+ 16us) server_negotiation.cc:407] Connection header received
0619 08:19:43.776511 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:43.776512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:43.776522 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:43.776595 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:43.776940 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:43.777566 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:43.778285 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:43.778507 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:43.778567 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:43.784684 (+ 6117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.129:46101: BlockingRecv error: recv got EOF from 127.4.151.129:46101 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
Jun 19 08:19:43 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:43 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:43.903703 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:43.898267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.130:57653)
0619 08:19:43.898498 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:43.898516 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:43.898628 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:43.898801 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:43.898802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:43.898990 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:43.899038 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:43.899040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:43.899735 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:43.899736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:43.900292 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:43.900293 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:43.900686 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:43.900687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:43.900744 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:43.901289 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:43.901293 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:43.903616 (+ 2323us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":106,"threads_started":1}
W20260619 08:19:43.903829 13932 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:43.898378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:57653 (local address 127.4.151.190:40109)
0619 08:19:43.898475 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:43.898478 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:43.898607 (+ 129us) server_negotiation.cc:407] Connection header received
0619 08:19:43.898691 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:43.898692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:43.898708 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:43.898776 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:43.899123 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:43.899642 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:43.900394 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:43.900597 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:43.900686 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:43.903725 (+ 3039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.130:57653: BlockingRecv error: recv got EOF from 127.4.151.130:57653 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 08:19:44 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:44 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.131@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:44.049206 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:44.043709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.131:40673)
0619 08:19:44.043857 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:44.043867 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:44.043920 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:44.044178 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:44.044179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:44.044373 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:44.044422 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:44.044424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:44.045245 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:44.045246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:44.045812 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:44.045814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:44.046221 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:44.046222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:44.046271 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:44.046887 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:44.046892 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:44.049144 (+ 2252us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
W20260619 08:19:44.049312 13932 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:44.043831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:40673 (local address 127.4.151.190:40109)
0619 08:19:44.043906 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:44.043907 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:44.043918 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:19:44.044037 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:44.044038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:44.044051 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:44.044113 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:44.044541 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:44.045125 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:44.045902 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:44.046137 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:44.046197 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:44.049241 (+ 3044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.131:40673: BlockingRecv error: recv got EOF from 127.4.151.131:40673 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 08:19:44 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:44 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.129@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:44.790475 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:44.785153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.129:53711)
0619 08:19:44.785336 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:44.785348 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:44.785488 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:44.785684 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:44.785685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:44.785859 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:44.785915 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:44.785916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:44.786749 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:44.786750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:44.787287 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:44.787288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:44.787676 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:44.787677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:44.787727 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:44.788199 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:44.788204 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:44.790321 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":93,"threads_started":1}
W20260619 08:19:44.790591 13936 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:44.785269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:53711 (local address 127.4.151.190:40109)
0619 08:19:44.785417 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:44.785419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:44.785477 (+ 58us) server_negotiation.cc:407] Connection header received
0619 08:19:44.785559 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:44.785560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:44.785570 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:44.785632 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:44.786126 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:44.786659 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:44.787375 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:44.787585 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:44.787647 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:44.790508 (+ 2861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.129:53711: BlockingRecv error: recv got EOF from 127.4.151.129:53711 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
Jun 19 08:19:44 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:19:44 dist-test-slave-d3n8 krb5kdc[12877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.151.130@KRBTEST.COM for oryx/127.4.151.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:19:44.910130 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:44.904451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40109 (local address 127.4.151.130:39599)
0619 08:19:44.904636 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:44.904650 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:44.904760 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:44.904983 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:44.904984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:44.905263 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:44.905326 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:44.905328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:44.906082 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:44.906083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:44.906626 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:44.906627 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:44.907011 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:44.907011 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:44.907092 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:44.907648 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:44.907653 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:44.910027 (+ 2374us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.151.190:40109: Server oryx/127.4.151.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":82,"threads_started":1}
W20260619 08:19:44.910326 13936 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:44.904499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:39599 (local address 127.4.151.190:40109)
0619 08:19:44.904649 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:44.904652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:44.904738 (+ 86us) server_negotiation.cc:407] Connection header received
0619 08:19:44.904851 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:44.904852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:44.904862 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:44.904977 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:44.905475 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:44.905986 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:44.906719 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:44.906923 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:44.907009 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:44.910209 (+ 3200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.151.130:39599: BlockingRecv error: recv got EOF from 127.4.151.130:39599 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
I20260619 08:19:44.966514 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 13460
I20260619 08:19:44.971038 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 13613
I20260619 08:19:44.975597 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 13745
I20260619 08:19:44.979820 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 13390
2026-06-19T08:19:44Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17601 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T08:19:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:19:44Z Disabled control of system clock
I20260619 08:19:45.008322 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:44557
--webserver_interface=127.4.151.190
--webserver_port=0
--builtin_ntp_servers=127.4.151.148:36729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:44557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 08:19:45.091284 13941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:45.091497 13941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:45.091518 13941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:45.092756 13941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:19:45.092798 13941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:45.092811 13941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:45.092824 13941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:19:45.092837 13941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:19:45.094376 13941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:36729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:44557
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:44557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:45.094594 13941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:45.094836 13941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:45.101300 13949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:45.101284 13946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:45.101312 13947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:45.101318 13941 server_base.cc:1034] running on GCE node
I20260619 08:19:45.102190 13941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:45.102622 13941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:45.103827 13941 hybrid_clock.cc:648] HybridClock initialized: now 1781857185103803 us; error 44 us; skew 500 ppm
I20260619 08:19:45.105327 13941 webserver.cc:460] Webserver started at http://127.4.151.190:44379/ using document root <none> and password file <none>
I20260619 08:19:45.105650 13941 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:45.105731 13941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:45.105893 13941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:45.107040 13941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "f7495b2fc5bc402196f52c6cb1dba6be"
format_stamp: "Formatted at 2026-06-19 08:19:45 on dist-test-slave-d3n8"
server_key: "7891bc0981734e526f37fba053babadd"
server_key_iv: "c5167e962ed7e5107bbe2af554bcc302"
server_key_version: "encryptionkey@0"
I20260619 08:19:45.107425 13941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "f7495b2fc5bc402196f52c6cb1dba6be"
format_stamp: "Formatted at 2026-06-19 08:19:45 on dist-test-slave-d3n8"
server_key: "7891bc0981734e526f37fba053babadd"
server_key_iv: "c5167e962ed7e5107bbe2af554bcc302"
server_key_version: "encryptionkey@0"
I20260619 08:19:45.108860 13941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:19:45.109812 13955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:45.110059 13941 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:45.110203 13941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "f7495b2fc5bc402196f52c6cb1dba6be"
format_stamp: "Formatted at 2026-06-19 08:19:45 on dist-test-slave-d3n8"
server_key: "7891bc0981734e526f37fba053babadd"
server_key_iv: "c5167e962ed7e5107bbe2af554bcc302"
server_key_version: "encryptionkey@0"
I20260619 08:19:45.110301 13941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:45.121156 13941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:45.121543 13941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:45.121701 13941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:45.125531 13941 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:44557
I20260619 08:19:45.125584 14007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:44557 every 8 connection(s)
I20260619 08:19:45.125958 13941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:19:45.126626 14008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:45.129496 14008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7495b2fc5bc402196f52c6cb1dba6be: Bootstrap starting.
I20260619 08:19:45.130119 14008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7495b2fc5bc402196f52c6cb1dba6be: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:45.130435 14008 log.cc:824] T 00000000000000000000000000000000 P f7495b2fc5bc402196f52c6cb1dba6be: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:45.131186 14008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7495b2fc5bc402196f52c6cb1dba6be: No bootstrap required, opened a new log
I20260619 08:19:45.132845 14008 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7495b2fc5bc402196f52c6cb1dba6be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7495b2fc5bc402196f52c6cb1dba6be" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44557 } }
I20260619 08:19:45.133015 14008 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7495b2fc5bc402196f52c6cb1dba6be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:45.133040 14008 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7495b2fc5bc402196f52c6cb1dba6be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7495b2fc5bc402196f52c6cb1dba6be, State: Initialized, Role: FOLLOWER
I20260619 08:19:45.133200 14008 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7495b2fc5bc402196f52c6cb1dba6be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7495b2fc5bc402196f52c6cb1dba6be" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44557 } }
I20260619 08:19:45.133235 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 13941
I20260619 08:19:45.133278 14008 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7495b2fc5bc402196f52c6cb1dba6be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:19:45.133303 14008 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7495b2fc5bc402196f52c6cb1dba6be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:19:45.133335 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:19:45.133390 14008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7495b2fc5bc402196f52c6cb1dba6be [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:45.133567 4702 external_mini_cluster.cc:1428] Setting key 52bb9623ab596478451dd18a799090f7
I20260619 08:19:45.134202 14008 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7495b2fc5bc402196f52c6cb1dba6be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7495b2fc5bc402196f52c6cb1dba6be" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44557 } }
I20260619 08:19:45.134326 14008 leader_election.cc:302] T 00000000000000000000000000000000 P f7495b2fc5bc402196f52c6cb1dba6be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7495b2fc5bc402196f52c6cb1dba6be; no voters:
I20260619 08:19:45.134505 14008 leader_election.cc:288] T 00000000000000000000000000000000 P f7495b2fc5bc402196f52c6cb1dba6be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:19:45.134533 14013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7495b2fc5bc402196f52c6cb1dba6be [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:45.134752 14013 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7495b2fc5bc402196f52c6cb1dba6be [term 1 LEADER]: Becoming Leader. State: Replica: f7495b2fc5bc402196f52c6cb1dba6be, State: Running, Role: LEADER
I20260619 08:19:45.134871 14013 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7495b2fc5bc402196f52c6cb1dba6be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7495b2fc5bc402196f52c6cb1dba6be" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44557 } }
I20260619 08:19:45.134745 14008 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7495b2fc5bc402196f52c6cb1dba6be [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:19:45.135227 14015 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7495b2fc5bc402196f52c6cb1dba6be [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7495b2fc5bc402196f52c6cb1dba6be. Latest consensus state: current_term: 1 leader_uuid: "f7495b2fc5bc402196f52c6cb1dba6be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7495b2fc5bc402196f52c6cb1dba6be" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44557 } } }
I20260619 08:19:45.135324 14015 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7495b2fc5bc402196f52c6cb1dba6be [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:45.135496 14014 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7495b2fc5bc402196f52c6cb1dba6be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7495b2fc5bc402196f52c6cb1dba6be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7495b2fc5bc402196f52c6cb1dba6be" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44557 } } }
I20260619 08:19:45.135591 14014 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7495b2fc5bc402196f52c6cb1dba6be [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:45.135741 14017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:19:45.136270 14017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:19:45.138134 14017 catalog_manager.cc:1347] Generated new cluster ID: aae95732fd874b139c5b08b8ac523b72
I20260619 08:19:45.138216 14017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:19:45.140375 14012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:45.134203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47880 (local address 127.4.151.190:44557)
0619 08:19:45.134454 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:45.134465 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:45.134501 (+ 36us) server_negotiation.cc:407] Connection header received
0619 08:19:45.134626 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:45.134634 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:45.134704 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:45.134853 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:45.135098 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:45.136995 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:45.138080 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:45.138709 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:45.138841 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:45.139616 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:45.139621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:45.139621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:19:45.139844 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:45.139905 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:19:45.139921 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:45.139999 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:45.140066 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 08:19:45.140166 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":109,"threads_started":1}
I20260619 08:19:45.150338 14017 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:19:45.150944 14017 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:19:45.155680 14017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7495b2fc5bc402196f52c6cb1dba6be: Generated new TSK 0
I20260619 08:19:45.155893 14017 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:19:45.160444 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:44557
--builtin_ntp_servers=127.4.151.148:36729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 08:19:45.244202 14032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:45.244436 14032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:45.244459 14032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:45.245836 14032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:45.245886 14032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:45.245913 14032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:19:45.247364 14032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:36729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:44557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:45.247654 14032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:45.247987 14032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:45.253218 14038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:45.253229 14037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:45.253433 14032 server_base.cc:1034] running on GCE node
W20260619 08:19:45.253245 14040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:45.253938 14032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:45.254253 14032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:45.255430 14032 hybrid_clock.cc:648] HybridClock initialized: now 1781857185255411 us; error 40 us; skew 500 ppm
I20260619 08:19:45.257064 14032 webserver.cc:460] Webserver started at http://127.4.151.129:45751/ using document root <none> and password file <none>
I20260619 08:19:45.257339 14032 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:45.257426 14032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:45.257579 14032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:45.258961 14032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "9dad979cf77947fea894789770b8ff33"
format_stamp: "Formatted at 2026-06-19 08:19:45 on dist-test-slave-d3n8"
server_key: "22329e97a7adca3360ab1dda250c91e3"
server_key_iv: "89e817cc1e0dc67b8943b7d60276cf3f"
server_key_version: "encryptionkey@0"
I20260619 08:19:45.259366 14032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "9dad979cf77947fea894789770b8ff33"
format_stamp: "Formatted at 2026-06-19 08:19:45 on dist-test-slave-d3n8"
server_key: "22329e97a7adca3360ab1dda250c91e3"
server_key_iv: "89e817cc1e0dc67b8943b7d60276cf3f"
server_key_version: "encryptionkey@0"
I20260619 08:19:45.261165 14032 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260619 08:19:45.262102 14046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:45.262300 14032 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:45.262387 14032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "9dad979cf77947fea894789770b8ff33"
format_stamp: "Formatted at 2026-06-19 08:19:45 on dist-test-slave-d3n8"
server_key: "22329e97a7adca3360ab1dda250c91e3"
server_key_iv: "89e817cc1e0dc67b8943b7d60276cf3f"
server_key_version: "encryptionkey@0"
I20260619 08:19:45.262459 14032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:45.287904 14032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:45.288211 14032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:45.288321 14032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:45.288545 14032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:45.288821 14032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:45.288849 14032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:45.288872 14032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:45.288885 14032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:45.295154 14032 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:42591
I20260619 08:19:45.295235 14159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:42591 every 8 connection(s)
I20260619 08:19:45.295569 14032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
I20260619 08:19:45.295759 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 14032
I20260619 08:19:45.295845 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:19:45.296073 4702 external_mini_cluster.cc:1428] Setting key 0818b4bd8d87e0194a8137f00f26bbc9
I20260619 08:19:45.297111 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:44557
--builtin_ntp_servers=127.4.151.148:36729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 08:19:45.302575 14012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:45.296436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:36915 (local address 127.4.151.190:44557)
0619 08:19:45.296496 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:45.296498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:45.296883 (+ 385us) server_negotiation.cc:407] Connection header received
0619 08:19:45.297724 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:45.297725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:45.297739 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:45.297796 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:45.298689 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:45.299306 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:45.300867 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:45.301222 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:45.301284 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:45.302298 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:45.302299 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:45.302299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:19:45.302306 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:45.302320 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:19:45.302325 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:45.302372 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:45.302469 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 08:19:45.302503 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 08:19:45.302899 14162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:45.296400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44557 (local address 127.4.151.129:36915)
0619 08:19:45.296614 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:45.296691 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:45.297608 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:45.297866 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:45.297870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:45.297880 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:45.298581 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:45.298587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:45.299384 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:45.299385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:45.300776 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:45.300780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:45.301317 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:45.301318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:45.301529 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:45.302128 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:19:45.302129 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:45.302171 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:19:45.302204 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:45.302376 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:45.302381 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:45.302384 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:19:45.302467 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 08:19:45.302612 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260619 08:19:45.303617 14160 heartbeater.cc:347] Connected to a master server at 127.4.151.190:44557
I20260619 08:19:45.303728 14160 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:45.303937 14160 heartbeater.cc:511] Master 127.4.151.190:44557 requested a full tablet report, sending...
I20260619 08:19:45.304499 13972 ts_manager.cc:194] Registered new tserver with Master: 9dad979cf77947fea894789770b8ff33 (127.4.151.129:42591)
I20260619 08:19:45.305183 13972 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.151.129:36915
W20260619 08:19:45.379954 14163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:45.380173 14163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:45.380195 14163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:45.381482 14163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:45.381531 14163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:45.381556 14163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:19:45.382936 14163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:36729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:44557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:45.383203 14163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:45.383518 14163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:45.388751 14169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:45.388890 14168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:45.388762 14171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:45.389117 14163 server_base.cc:1034] running on GCE node
I20260619 08:19:45.389415 14163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:45.389714 14163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:45.390888 14163 hybrid_clock.cc:648] HybridClock initialized: now 1781857185390872 us; error 43 us; skew 500 ppm
I20260619 08:19:45.392643 14163 webserver.cc:460] Webserver started at http://127.4.151.130:46457/ using document root <none> and password file <none>
I20260619 08:19:45.392956 14163 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:45.393010 14163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:45.393239 14163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:45.394532 14163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "5020822f4b804450a0a42427cfc62f4a"
format_stamp: "Formatted at 2026-06-19 08:19:45 on dist-test-slave-d3n8"
server_key: "557bbcfe2227e76e60e179c5ada7d61f"
server_key_iv: "c1e6fa1f0262179c161065a9985a6b29"
server_key_version: "encryptionkey@0"
I20260619 08:19:45.394934 14163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "5020822f4b804450a0a42427cfc62f4a"
format_stamp: "Formatted at 2026-06-19 08:19:45 on dist-test-slave-d3n8"
server_key: "557bbcfe2227e76e60e179c5ada7d61f"
server_key_iv: "c1e6fa1f0262179c161065a9985a6b29"
server_key_version: "encryptionkey@0"
I20260619 08:19:45.396409 14163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:19:45.397270 14177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:45.397508 14163 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:45.397595 14163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "5020822f4b804450a0a42427cfc62f4a"
format_stamp: "Formatted at 2026-06-19 08:19:45 on dist-test-slave-d3n8"
server_key: "557bbcfe2227e76e60e179c5ada7d61f"
server_key_iv: "c1e6fa1f0262179c161065a9985a6b29"
server_key_version: "encryptionkey@0"
I20260619 08:19:45.397645 14163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:45.418303 14163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:45.418589 14163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:45.418694 14163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:45.418880 14163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:45.419160 14163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:45.419193 14163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:45.419217 14163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:45.419236 14163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:45.424865 14163 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:39279
I20260619 08:19:45.425027 14290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:39279 every 8 connection(s)
I20260619 08:19:45.425307 14163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
I20260619 08:19:45.430989 14012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:45.426288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:40267 (local address 127.4.151.190:44557)
0619 08:19:45.426363 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:45.426366 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:45.426582 (+ 216us) server_negotiation.cc:407] Connection header received
0619 08:19:45.427146 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:45.427148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:45.427161 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:45.427212 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:45.428044 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:45.428561 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:45.429605 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:45.429798 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:45.429879 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:45.430676 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:45.430677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:45.430678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:19:45.430683 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:45.430699 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:19:45.430703 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:45.430775 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:45.430871 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 08:19:45.430900 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 08:19:45.431209 14293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:45.426217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44557 (local address 127.4.151.130:40267)
0619 08:19:45.426457 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:45.426492 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:45.427028 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:45.427286 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:45.427289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:45.427297 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:45.427830 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:45.427835 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:45.428655 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:45.428656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:45.429509 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:45.429511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:45.429889 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:45.429890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:45.430029 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:45.430524 (+ 495us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:19:45.430524 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:45.430551 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:19:45.430573 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:45.430776 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:45.430782 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:45.430785 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:19:45.430872 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 08:19:45.430972 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
I20260619 08:19:45.431741 14291 heartbeater.cc:347] Connected to a master server at 127.4.151.190:44557
I20260619 08:19:45.431821 14291 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:45.432039 14291 heartbeater.cc:511] Master 127.4.151.190:44557 requested a full tablet report, sending...
I20260619 08:19:45.432492 13972 ts_manager.cc:194] Registered new tserver with Master: 5020822f4b804450a0a42427cfc62f4a (127.4.151.130:39279)
I20260619 08:19:45.432847 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 14163
I20260619 08:19:45.432932 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:19:45.433127 4702 external_mini_cluster.cc:1428] Setting key 7f5196d4080dcd444acb53ef878dfc35
I20260619 08:19:45.433113 13972 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.151.130:40267
I20260619 08:19:45.434154 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:44557
--builtin_ntp_servers=127.4.151.148:36729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 08:19:45.517124 14294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:45.517334 14294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:45.517355 14294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:45.518551 14294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:45.518596 14294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:45.518620 14294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:19:45.519995 14294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:36729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:44557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:45.520192 14294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:45.520443 14294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:45.525588 14299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:45.525619 14300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:45.525591 14302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:45.526068 14294 server_base.cc:1034] running on GCE node
I20260619 08:19:45.526288 14294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:45.526531 14294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:45.527683 14294 hybrid_clock.cc:648] HybridClock initialized: now 1781857185527667 us; error 32 us; skew 500 ppm
I20260619 08:19:45.529078 14294 webserver.cc:460] Webserver started at http://127.4.151.131:41451/ using document root <none> and password file <none>
I20260619 08:19:45.529318 14294 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:45.529402 14294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:45.529580 14294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:45.530682 14294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "e570bc7f5b0f4dd4be24ed0ee0ee10a3"
format_stamp: "Formatted at 2026-06-19 08:19:45 on dist-test-slave-d3n8"
server_key: "01ed87a9b488cd8292084e11b7d7c8ca"
server_key_iv: "4805b6241cdc0441ce31a47d1787ae18"
server_key_version: "encryptionkey@0"
I20260619 08:19:45.531062 14294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "e570bc7f5b0f4dd4be24ed0ee0ee10a3"
format_stamp: "Formatted at 2026-06-19 08:19:45 on dist-test-slave-d3n8"
server_key: "01ed87a9b488cd8292084e11b7d7c8ca"
server_key_iv: "4805b6241cdc0441ce31a47d1787ae18"
server_key_version: "encryptionkey@0"
I20260619 08:19:45.532402 14294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:19:45.533232 14308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:45.533493 14294 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:45.533612 14294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "e570bc7f5b0f4dd4be24ed0ee0ee10a3"
format_stamp: "Formatted at 2026-06-19 08:19:45 on dist-test-slave-d3n8"
server_key: "01ed87a9b488cd8292084e11b7d7c8ca"
server_key_iv: "4805b6241cdc0441ce31a47d1787ae18"
server_key_version: "encryptionkey@0"
I20260619 08:19:45.533689 14294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:45.541668 14294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:45.541961 14294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:45.542107 14294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:45.542323 14294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:45.542654 14294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:45.542711 14294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:45.542742 14294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:45.542770 14294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:45.548678 14294 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:38909
I20260619 08:19:45.548723 14421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:38909 every 8 connection(s)
I20260619 08:19:45.549078 14294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
I20260619 08:19:45.549335 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 14294
I20260619 08:19:45.549543 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:19:45.549786 4702 external_mini_cluster.cc:1428] Setting key 2bc7ad839ea2e7a8b822643b9dfde2e0
I20260619 08:19:45.556154 14012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:45.549930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:34531 (local address 127.4.151.190:44557)
0619 08:19:45.549979 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:45.549981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:45.550253 (+ 272us) server_negotiation.cc:407] Connection header received
0619 08:19:45.550895 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:45.550897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:45.550909 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:45.550953 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:45.551860 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:45.552428 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:45.554142 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:45.554531 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:45.554607 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:45.555747 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:45.555748 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:45.555749 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:19:45.555755 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:45.555787 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:19:45.555792 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:45.555851 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:45.556005 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 08:19:45.556038 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:19:45.556344 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:45.549930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44557 (local address 127.4.151.131:34531)
0619 08:19:45.550141 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:45.550181 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:45.550809 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:45.551021 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:45.551025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:45.551036 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:45.551769 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:45.551775 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:45.552502 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:45.552503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:45.553995 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:45.553999 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:45.554627 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:45.554628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:45.554809 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:45.555549 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:19:45.555550 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:45.555590 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:19:45.555619 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:45.555875 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:45.555880 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:45.555884 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:19:45.555990 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 08:19:45.556128 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":97,"threads_started":1}
I20260619 08:19:45.557524 14422 heartbeater.cc:347] Connected to a master server at 127.4.151.190:44557
I20260619 08:19:45.557662 14422 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:45.557848 14422 heartbeater.cc:511] Master 127.4.151.190:44557 requested a full tablet report, sending...
I20260619 08:19:45.558288 13972 ts_manager.cc:194] Registered new tserver with Master: e570bc7f5b0f4dd4be24ed0ee0ee10a3 (127.4.151.131:38909)
I20260619 08:19:45.558785 13972 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.151.131:34531
I20260619 08:19:45.561671 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 08:19:45.563560 14012 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:45.563139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47886 (local address 127.4.151.190:44557)
0619 08:19:45.563221 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:45.563223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:45.563232 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:19:45.563294 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:45.563295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:45.563306 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:45.563356 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:45.563491 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47886: BlockingRecv error: recv got EOF from 127.0.0.1:47886 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 08:19:45.563628 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 14032
I20260619 08:19:45.568442 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 14163
I20260619 08:19:45.572476 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 14294
I20260619 08:19:45.576776 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 13941
2026-06-19T08:19:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (595 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T08:19:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:19:45Z Disabled control of system clock
I20260619 08:19:45.604820 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:34237
--webserver_interface=127.4.151.190
--webserver_port=0
--builtin_ntp_servers=127.4.151.148:37327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:34237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 08:19:45.694571 14433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:45.694854 14433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:45.694896 14433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:45.696322 14433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:19:45.696398 14433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:45.696424 14433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:45.696460 14433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:19:45.696489 14433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:19:45.698097 14433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:37327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:34237
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:34237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:45.698413 14433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:45.698745 14433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:45.704329 14439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:45.704335 14438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:45.704349 14441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:45.704587 14433 server_base.cc:1034] running on GCE node
I20260619 08:19:45.704838 14433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:45.705180 14433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:45.706359 14433 hybrid_clock.cc:648] HybridClock initialized: now 1781857185706334 us; error 57 us; skew 500 ppm
I20260619 08:19:45.707690 14433 webserver.cc:460] Webserver started at http://127.4.151.190:37817/ using document root <none> and password file <none>
I20260619 08:19:45.707935 14433 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:45.708005 14433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:45.708210 14433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:45.709313 14433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "1926a86d7e2e41c9bb1dce9732a6d7c2"
format_stamp: "Formatted at 2026-06-19 08:19:45 on dist-test-slave-d3n8"
server_key: "4a6022c58115fdb14a07137484dfa187"
server_key_iv: "76b72d11a312abf17e3625d441d24684"
server_key_version: "encryptionkey@0"
I20260619 08:19:45.709810 14433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "1926a86d7e2e41c9bb1dce9732a6d7c2"
format_stamp: "Formatted at 2026-06-19 08:19:45 on dist-test-slave-d3n8"
server_key: "4a6022c58115fdb14a07137484dfa187"
server_key_iv: "76b72d11a312abf17e3625d441d24684"
server_key_version: "encryptionkey@0"
I20260619 08:19:45.711409 14433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:19:45.712313 14447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:45.712508 14433 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:45.712626 14433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "1926a86d7e2e41c9bb1dce9732a6d7c2"
format_stamp: "Formatted at 2026-06-19 08:19:45 on dist-test-slave-d3n8"
server_key: "4a6022c58115fdb14a07137484dfa187"
server_key_iv: "76b72d11a312abf17e3625d441d24684"
server_key_version: "encryptionkey@0"
I20260619 08:19:45.712710 14433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:45.716535 14433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:45.716850 14433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:45.717000 14433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:45.721112 14433 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:34237
I20260619 08:19:45.721140 14499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:34237 every 8 connection(s)
I20260619 08:19:45.721580 14433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:19:45.722232 14500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:45.725037 14500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1926a86d7e2e41c9bb1dce9732a6d7c2: Bootstrap starting.
I20260619 08:19:45.725682 14500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1926a86d7e2e41c9bb1dce9732a6d7c2: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:45.725921 14500 log.cc:824] T 00000000000000000000000000000000 P 1926a86d7e2e41c9bb1dce9732a6d7c2: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:45.726598 14500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1926a86d7e2e41c9bb1dce9732a6d7c2: No bootstrap required, opened a new log
I20260619 08:19:45.728174 14500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1926a86d7e2e41c9bb1dce9732a6d7c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1926a86d7e2e41c9bb1dce9732a6d7c2" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 34237 } }
I20260619 08:19:45.728317 14500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1926a86d7e2e41c9bb1dce9732a6d7c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:45.728338 14500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1926a86d7e2e41c9bb1dce9732a6d7c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1926a86d7e2e41c9bb1dce9732a6d7c2, State: Initialized, Role: FOLLOWER
I20260619 08:19:45.728480 14500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1926a86d7e2e41c9bb1dce9732a6d7c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1926a86d7e2e41c9bb1dce9732a6d7c2" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 34237 } }
I20260619 08:19:45.728569 14500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1926a86d7e2e41c9bb1dce9732a6d7c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:19:45.728591 14500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1926a86d7e2e41c9bb1dce9732a6d7c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:19:45.728619 14500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1926a86d7e2e41c9bb1dce9732a6d7c2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:45.729120 14500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1926a86d7e2e41c9bb1dce9732a6d7c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1926a86d7e2e41c9bb1dce9732a6d7c2" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 34237 } }
I20260619 08:19:45.729202 14500 leader_election.cc:302] T 00000000000000000000000000000000 P 1926a86d7e2e41c9bb1dce9732a6d7c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1926a86d7e2e41c9bb1dce9732a6d7c2; no voters:
I20260619 08:19:45.729339 14500 leader_election.cc:288] T 00000000000000000000000000000000 P 1926a86d7e2e41c9bb1dce9732a6d7c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:19:45.729431 14503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1926a86d7e2e41c9bb1dce9732a6d7c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:45.729696 14500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1926a86d7e2e41c9bb1dce9732a6d7c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:19:45.729724 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 14433
I20260619 08:19:45.729800 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:19:45.729679 14503 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1926a86d7e2e41c9bb1dce9732a6d7c2 [term 1 LEADER]: Becoming Leader. State: Replica: 1926a86d7e2e41c9bb1dce9732a6d7c2, State: Running, Role: LEADER
I20260619 08:19:45.729902 14503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1926a86d7e2e41c9bb1dce9732a6d7c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1926a86d7e2e41c9bb1dce9732a6d7c2" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 34237 } }
I20260619 08:19:45.730023 4702 external_mini_cluster.cc:1428] Setting key 604a08efab3fd79b602d395eaef58bad
I20260619 08:19:45.730334 14505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1926a86d7e2e41c9bb1dce9732a6d7c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1926a86d7e2e41c9bb1dce9732a6d7c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1926a86d7e2e41c9bb1dce9732a6d7c2" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 34237 } } }
I20260619 08:19:45.730507 14505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1926a86d7e2e41c9bb1dce9732a6d7c2 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:45.730568 14504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1926a86d7e2e41c9bb1dce9732a6d7c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1926a86d7e2e41c9bb1dce9732a6d7c2. Latest consensus state: current_term: 1 leader_uuid: "1926a86d7e2e41c9bb1dce9732a6d7c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1926a86d7e2e41c9bb1dce9732a6d7c2" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 34237 } } }
I20260619 08:19:45.730729 14504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1926a86d7e2e41c9bb1dce9732a6d7c2 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:45.730791 14513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:19:45.731523 14513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:19:45.733035 14514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:45.730757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51310 (local address 127.4.151.190:34237)
0619 08:19:45.730967 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:45.730977 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:45.731018 (+ 41us) server_negotiation.cc:407] Connection header received
0619 08:19:45.731215 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:45.731224 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:45.731289 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:45.731702 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:45.732272 (+ 570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:45.732276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:45.732276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:19:45.732280 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:45.732321 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:19:45.732335 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:45.732425 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:45.732715 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 08:19:45.732782 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":83,"threads_started":1}
I20260619 08:19:45.733847 14513 catalog_manager.cc:1347] Generated new cluster ID: 2c2c89cfcf35417f9c8e55fce59af4c3
I20260619 08:19:45.733917 14513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:19:45.743321 14513 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:19:45.743568 14513 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:19:45.749097 14513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1926a86d7e2e41c9bb1dce9732a6d7c2: Generated new TSK 0
I20260619 08:19:45.749327 14513 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:19:45.757608 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:34237
--builtin_ntp_servers=127.4.151.148:37327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 08:19:45.840560 14524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:45.840822 14524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:45.840862 14524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:45.842159 14524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:45.842227 14524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:45.842272 14524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:19:45.843652 14524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:37327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:34237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:45.843920 14524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:45.844213 14524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:45.850359 14530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:45.850353 14532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:45.850358 14529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:45.850767 14524 server_base.cc:1034] running on GCE node
I20260619 08:19:45.850983 14524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:45.851338 14524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:45.852535 14524 hybrid_clock.cc:648] HybridClock initialized: now 1781857185852514 us; error 55 us; skew 500 ppm
I20260619 08:19:45.854118 14524 webserver.cc:460] Webserver started at http://127.4.151.129:40341/ using document root <none> and password file <none>
I20260619 08:19:45.854378 14524 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:45.854432 14524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:45.854555 14524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:45.855937 14524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "9f3edf1eac07409b8a34c467b8fb7077"
format_stamp: "Formatted at 2026-06-19 08:19:45 on dist-test-slave-d3n8"
server_key: "8e2ae70309f55c4353b690cc5d64b829"
server_key_iv: "0185767aaabd2ecb9a1ba03140109d1e"
server_key_version: "encryptionkey@0"
I20260619 08:19:45.856338 14524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "9f3edf1eac07409b8a34c467b8fb7077"
format_stamp: "Formatted at 2026-06-19 08:19:45 on dist-test-slave-d3n8"
server_key: "8e2ae70309f55c4353b690cc5d64b829"
server_key_iv: "0185767aaabd2ecb9a1ba03140109d1e"
server_key_version: "encryptionkey@0"
I20260619 08:19:45.858014 14524 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260619 08:19:45.858924 14538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:45.859175 14524 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:45.859290 14524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "9f3edf1eac07409b8a34c467b8fb7077"
format_stamp: "Formatted at 2026-06-19 08:19:45 on dist-test-slave-d3n8"
server_key: "8e2ae70309f55c4353b690cc5d64b829"
server_key_iv: "0185767aaabd2ecb9a1ba03140109d1e"
server_key_version: "encryptionkey@0"
I20260619 08:19:45.859372 14524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:45.864044 14524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:45.864286 14524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:45.864383 14524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:45.864547 14524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:45.864805 14524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:45.864835 14524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:45.864857 14524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:45.864869 14524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:45.870857 14524 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:38993
I20260619 08:19:45.870925 14651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:38993 every 8 connection(s)
I20260619 08:19:45.871245 14524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
I20260619 08:19:45.872726 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 14524
I20260619 08:19:45.872836 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:19:45.873035 4702 external_mini_cluster.cc:1428] Setting key a400cd2923df7669799cbae6774e9203
I20260619 08:19:45.874256 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:34237
--builtin_ntp_servers=127.4.151.148:37327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 08:19:45.874706 14514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:45.871994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:45101 (local address 127.4.151.190:34237)
0619 08:19:45.872039 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:45.872041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:45.872287 (+ 246us) server_negotiation.cc:407] Connection header received
0619 08:19:45.872792 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:45.872794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:45.872804 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:45.872885 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:45.874370 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:45.874371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:45.874371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:19:45.874377 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:45.874400 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:19:45.874405 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:45.874479 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:45.874618 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 08:19:45.874636 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:19:45.874903 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:45.871985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:34237 (local address 127.4.151.129:45101)
0619 08:19:45.872158 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:45.872191 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:45.872695 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:45.872937 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:45.872940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:45.872947 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:45.874181 (+ 1234us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:19:45.874182 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:45.874221 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:19:45.874259 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:45.874507 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:45.874514 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:45.874517 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:19:45.874614 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 08:19:45.874679 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260619 08:19:45.875370 14652 heartbeater.cc:347] Connected to a master server at 127.4.151.190:34237
I20260619 08:19:45.875514 14652 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:45.875689 14652 heartbeater.cc:511] Master 127.4.151.190:34237 requested a full tablet report, sending...
I20260619 08:19:45.876278 14464 ts_manager.cc:194] Registered new tserver with Master: 9f3edf1eac07409b8a34c467b8fb7077 (127.4.151.129:38993)
W20260619 08:19:45.955536 14655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:45.955788 14655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:45.955825 14655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:45.957105 14655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:45.957185 14655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:45.957228 14655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:19:45.958630 14655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:37327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:34237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:45.958902 14655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:45.959206 14655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:45.964560 14661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:45.964632 14655 server_base.cc:1034] running on GCE node
W20260619 08:19:45.964702 14660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:45.964560 14663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:45.965019 14655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:45.965355 14655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:45.966557 14655 hybrid_clock.cc:648] HybridClock initialized: now 1781857185966534 us; error 38 us; skew 500 ppm
I20260619 08:19:45.967867 14655 webserver.cc:460] Webserver started at http://127.4.151.130:45573/ using document root <none> and password file <none>
I20260619 08:19:45.968132 14655 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:45.968204 14655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:45.968355 14655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:45.969627 14655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "74d0cfb72a234947a62731b820fc5590"
format_stamp: "Formatted at 2026-06-19 08:19:45 on dist-test-slave-d3n8"
server_key: "ae3ac8b1556099998407628aa3653a28"
server_key_iv: "9268b088dabc646de21c3f2bedfb30f7"
server_key_version: "encryptionkey@0"
I20260619 08:19:45.969978 14655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "74d0cfb72a234947a62731b820fc5590"
format_stamp: "Formatted at 2026-06-19 08:19:45 on dist-test-slave-d3n8"
server_key: "ae3ac8b1556099998407628aa3653a28"
server_key_iv: "9268b088dabc646de21c3f2bedfb30f7"
server_key_version: "encryptionkey@0"
I20260619 08:19:45.971283 14655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 08:19:45.972090 14669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:45.972281 14655 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:45.972383 14655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "74d0cfb72a234947a62731b820fc5590"
format_stamp: "Formatted at 2026-06-19 08:19:45 on dist-test-slave-d3n8"
server_key: "ae3ac8b1556099998407628aa3653a28"
server_key_iv: "9268b088dabc646de21c3f2bedfb30f7"
server_key_version: "encryptionkey@0"
I20260619 08:19:45.972457 14655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:45.975983 14655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:45.976253 14655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:45.976395 14655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:45.976596 14655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:45.976912 14655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:45.976966 14655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:45.977006 14655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:45.977034 14655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:45.982798 14655 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:38783
I20260619 08:19:45.982837 14782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:38783 every 8 connection(s)
I20260619 08:19:45.983194 14655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
I20260619 08:19:45.986136 14514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:45.983940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:57037 (local address 127.4.151.190:34237)
0619 08:19:45.983984 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:45.983986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:45.984219 (+ 233us) server_negotiation.cc:407] Connection header received
0619 08:19:45.984664 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:45.984665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:45.984672 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:45.984733 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:45.985894 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:45.985896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:45.985896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:19:45.985900 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:45.985914 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:19:45.985919 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:45.985981 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:45.986069 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 08:19:45.986082 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 08:19:45.986388 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:45.983907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:34237 (local address 127.4.151.130:57037)
0619 08:19:45.984091 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:45.984130 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:45.984573 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:45.984791 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:45.984794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:45.984801 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:45.985753 (+ 952us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:19:45.985754 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:45.985781 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:19:45.985807 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:45.985975 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:45.985978 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:45.985980 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:19:45.986026 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 08:19:45.986159 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":98,"threads_started":1}
I20260619 08:19:45.986920 14783 heartbeater.cc:347] Connected to a master server at 127.4.151.190:34237
I20260619 08:19:45.987040 14783 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:45.987190 14783 heartbeater.cc:511] Master 127.4.151.190:34237 requested a full tablet report, sending...
I20260619 08:19:45.987545 14464 ts_manager.cc:194] Registered new tserver with Master: 74d0cfb72a234947a62731b820fc5590 (127.4.151.130:38783)
I20260619 08:19:45.989545 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 14655
I20260619 08:19:45.989658 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:19:45.989842 4702 external_mini_cluster.cc:1428] Setting key 8410e29b7f4ab3b3ae2d48a0894f1002
I20260619 08:19:45.990738 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:34237
--builtin_ntp_servers=127.4.151.148:37327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 08:19:46.073187 14786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:46.073416 14786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:46.073439 14786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:46.074646 14786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:46.074693 14786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:46.074719 14786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:19:46.076208 14786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:37327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:34237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:46.076471 14786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:46.076761 14786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:19:46.081835 14786 server_base.cc:1034] running on GCE node
W20260619 08:19:46.081768 14794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:46.081845 14791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:46.081777 14792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:46.082294 14786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:46.082558 14786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:46.083797 14786 hybrid_clock.cc:648] HybridClock initialized: now 1781857186083797 us; error 45 us; skew 500 ppm
I20260619 08:19:46.085232 14786 webserver.cc:460] Webserver started at http://127.4.151.131:35083/ using document root <none> and password file <none>
I20260619 08:19:46.085567 14786 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:46.085641 14786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:46.085783 14786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:46.086876 14786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "8608241eccbc46d3aedff864c3b949b1"
format_stamp: "Formatted at 2026-06-19 08:19:46 on dist-test-slave-d3n8"
server_key: "e33edf5c54f7c39842bf799bcb3c6128"
server_key_iv: "d7d0e24f7ed3c1c9e7c852fad4711814"
server_key_version: "encryptionkey@0"
I20260619 08:19:46.087265 14786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "8608241eccbc46d3aedff864c3b949b1"
format_stamp: "Formatted at 2026-06-19 08:19:46 on dist-test-slave-d3n8"
server_key: "e33edf5c54f7c39842bf799bcb3c6128"
server_key_iv: "d7d0e24f7ed3c1c9e7c852fad4711814"
server_key_version: "encryptionkey@0"
I20260619 08:19:46.088593 14786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:19:46.089610 14800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:46.089808 14786 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:46.089916 14786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "8608241eccbc46d3aedff864c3b949b1"
format_stamp: "Formatted at 2026-06-19 08:19:46 on dist-test-slave-d3n8"
server_key: "e33edf5c54f7c39842bf799bcb3c6128"
server_key_iv: "d7d0e24f7ed3c1c9e7c852fad4711814"
server_key_version: "encryptionkey@0"
I20260619 08:19:46.090000 14786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:46.093418 14786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:46.093696 14786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:46.093835 14786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:46.094070 14786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:46.094377 14786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:46.094430 14786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:46.094470 14786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:46.094496 14786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:46.100113 14786 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:41315
I20260619 08:19:46.100152 14913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:41315 every 8 connection(s)
I20260619 08:19:46.100490 14786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
I20260619 08:19:46.103564 14514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:46.101226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:57839 (local address 127.4.151.190:34237)
0619 08:19:46.101295 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:46.101297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:46.101604 (+ 307us) server_negotiation.cc:407] Connection header received
0619 08:19:46.102066 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:46.102067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:46.102072 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:46.102126 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:46.103330 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:46.103331 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:46.103331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:19:46.103336 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:46.103350 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:19:46.103355 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:46.103446 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:46.103496 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 08:19:46.103509 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 08:19:46.103787 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:46.101220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:34237 (local address 127.4.151.131:57839)
0619 08:19:46.101460 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:46.101495 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:46.101982 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:46.102175 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:46.102179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:46.102187 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:46.103168 (+ 981us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:19:46.103169 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:46.103196 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:19:46.103223 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:46.103447 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:46.103449 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:46.103451 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:19:46.103501 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 08:19:46.103554 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20260619 08:19:46.104283 14914 heartbeater.cc:347] Connected to a master server at 127.4.151.190:34237
I20260619 08:19:46.104380 14914 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:46.104524 14914 heartbeater.cc:511] Master 127.4.151.190:34237 requested a full tablet report, sending...
I20260619 08:19:46.104952 14464 ts_manager.cc:194] Registered new tserver with Master: 8608241eccbc46d3aedff864c3b949b1 (127.4.151.131:41315)
I20260619 08:19:46.105859 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 14786
I20260619 08:19:46.105966 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:19:46.106184 4702 external_mini_cluster.cc:1428] Setting key c914f5767edde9b2689553b1e1164b02
I20260619 08:19:46.106987 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 08:19:46.108729 14514 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:19:46.108298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51318 (local address 127.4.151.190:34237)
0619 08:19:46.108332 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:46.108334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:46.108438 (+ 104us) server_negotiation.cc:407] Connection header received
0619 08:19:46.108509 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:46.108510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:46.108514 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:46.108567 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:46.108653 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51318: BlockingRecv error: recv got EOF from 127.0.0.1:51318 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 08:19:46.108844 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 14524
I20260619 08:19:46.113595 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 14655
I20260619 08:19:46.117975 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 14786
I20260619 08:19:46.122463 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 14433
2026-06-19T08:19:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (545 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:19:46 dist-test-slave-d3n8 krb5kdc[14924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:19:46 dist-test-slave-d3n8 krb5kdc[14924](info): set up 2 sockets
Jun 19 08:19:46 dist-test-slave-d3n8 krb5kdc[14924](info): commencing operation
krb5kdc: starting...
W20260619 08:19:47.174454 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:19:47 dist-test-slave-d3n8 krb5kdc[14924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:19:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:19:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:19:47.301451 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:32861
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:44459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:32861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:47.384608 14937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:47.384833 14937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:47.384856 14937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:47.386183 14937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:19:47.386229 14937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:47.386243 14937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:47.386255 14937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:19:47.386268 14937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:19:47.387789 14937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:44459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:32861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:32861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:47.388060 14937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:47.388299 14937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:47.393497 14942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:47.393657 14937 server_base.cc:1034] running on GCE node
W20260619 08:19:47.393518 14943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:47.393493 14945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:47.394043 14937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:47.394352 14937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:47.395526 14937 hybrid_clock.cc:648] HybridClock initialized: now 1781857187395520 us; error 44 us; skew 500 ppm
Jun 19 08:19:47 dist-test-slave-d3n8 krb5kdc[14924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857187, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:47.398267 14937 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:19:47.398708 14937 webserver.cc:460] Webserver started at http://127.4.151.190:44783/ using document root <none> and password file <none>
I20260619 08:19:47.398957 14937 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:47.399034 14937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:47.399181 14937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:47.400274 14937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "df1e09cccc164c8f8418cce3cbc6a780"
format_stamp: "Formatted at 2026-06-19 08:19:47 on dist-test-slave-d3n8"
server_key: "f7ed12e1b70eef4bf037110a2cc9d6a2"
server_key_iv: "9ca9eb05ef05b7a7eb0a54437087b841"
server_key_version: "encryptionkey@0"
I20260619 08:19:47.400642 14937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "df1e09cccc164c8f8418cce3cbc6a780"
format_stamp: "Formatted at 2026-06-19 08:19:47 on dist-test-slave-d3n8"
server_key: "f7ed12e1b70eef4bf037110a2cc9d6a2"
server_key_iv: "9ca9eb05ef05b7a7eb0a54437087b841"
server_key_version: "encryptionkey@0"
I20260619 08:19:47.402155 14937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:19:47.402956 14952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:47.403165 14937 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:19:47.403257 14937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "df1e09cccc164c8f8418cce3cbc6a780"
format_stamp: "Formatted at 2026-06-19 08:19:47 on dist-test-slave-d3n8"
server_key: "f7ed12e1b70eef4bf037110a2cc9d6a2"
server_key_iv: "9ca9eb05ef05b7a7eb0a54437087b841"
server_key_version: "encryptionkey@0"
I20260619 08:19:47.403344 14937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:47.420543 14937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:47.423182 14937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:47.423362 14937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:47.427381 14937 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:32861
I20260619 08:19:47.427412 15004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:32861 every 8 connection(s)
I20260619 08:19:47.427755 14937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:19:47.428690 15005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:47.431186 15005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df1e09cccc164c8f8418cce3cbc6a780: Bootstrap starting.
I20260619 08:19:47.431793 15005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df1e09cccc164c8f8418cce3cbc6a780: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:47.432091 15005 log.cc:824] T 00000000000000000000000000000000 P df1e09cccc164c8f8418cce3cbc6a780: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:47.432823 15005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df1e09cccc164c8f8418cce3cbc6a780: No bootstrap required, opened a new log
I20260619 08:19:47.434535 15005 raft_consensus.cc:348] T 00000000000000000000000000000000 P df1e09cccc164c8f8418cce3cbc6a780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1e09cccc164c8f8418cce3cbc6a780" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 32861 } }
I20260619 08:19:47.434700 15005 raft_consensus.cc:374] T 00000000000000000000000000000000 P df1e09cccc164c8f8418cce3cbc6a780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:47.434748 15005 raft_consensus.cc:729] T 00000000000000000000000000000000 P df1e09cccc164c8f8418cce3cbc6a780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1e09cccc164c8f8418cce3cbc6a780, State: Initialized, Role: FOLLOWER
I20260619 08:19:47.434922 15005 consensus_queue.cc:260] T 00000000000000000000000000000000 P df1e09cccc164c8f8418cce3cbc6a780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1e09cccc164c8f8418cce3cbc6a780" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 32861 } }
I20260619 08:19:47.435025 15005 raft_consensus.cc:388] T 00000000000000000000000000000000 P df1e09cccc164c8f8418cce3cbc6a780 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:19:47.435070 15005 raft_consensus.cc:482] T 00000000000000000000000000000000 P df1e09cccc164c8f8418cce3cbc6a780 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:19:47.435129 15005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df1e09cccc164c8f8418cce3cbc6a780 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:47.435940 15005 raft_consensus.cc:504] T 00000000000000000000000000000000 P df1e09cccc164c8f8418cce3cbc6a780 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1e09cccc164c8f8418cce3cbc6a780" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 32861 } }
I20260619 08:19:47.436064 15005 leader_election.cc:302] T 00000000000000000000000000000000 P df1e09cccc164c8f8418cce3cbc6a780 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df1e09cccc164c8f8418cce3cbc6a780; no voters:
I20260619 08:19:47.436255 15005 leader_election.cc:288] T 00000000000000000000000000000000 P df1e09cccc164c8f8418cce3cbc6a780 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:19:47.436288 15008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df1e09cccc164c8f8418cce3cbc6a780 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:47.436395 15008 raft_consensus.cc:686] T 00000000000000000000000000000000 P df1e09cccc164c8f8418cce3cbc6a780 [term 1 LEADER]: Becoming Leader. State: Replica: df1e09cccc164c8f8418cce3cbc6a780, State: Running, Role: LEADER
I20260619 08:19:47.436479 15008 consensus_queue.cc:237] T 00000000000000000000000000000000 P df1e09cccc164c8f8418cce3cbc6a780 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1e09cccc164c8f8418cce3cbc6a780" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 32861 } }
I20260619 08:19:47.436547 15005 sys_catalog.cc:564] T 00000000000000000000000000000000 P df1e09cccc164c8f8418cce3cbc6a780 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:19:47.436671 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 14937
I20260619 08:19:47.436774 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:19:47.436846 15009 sys_catalog.cc:455] T 00000000000000000000000000000000 P df1e09cccc164c8f8418cce3cbc6a780 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df1e09cccc164c8f8418cce3cbc6a780" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1e09cccc164c8f8418cce3cbc6a780" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 32861 } } }
I20260619 08:19:47.436926 15009 sys_catalog.cc:458] T 00000000000000000000000000000000 P df1e09cccc164c8f8418cce3cbc6a780 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:47.437016 4702 external_mini_cluster.cc:1428] Setting key ddc738cb9d24c561da1d3b2006e3fc88
I20260619 08:19:47.437093 15010 sys_catalog.cc:455] T 00000000000000000000000000000000 P df1e09cccc164c8f8418cce3cbc6a780 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df1e09cccc164c8f8418cce3cbc6a780. Latest consensus state: current_term: 1 leader_uuid: "df1e09cccc164c8f8418cce3cbc6a780" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1e09cccc164c8f8418cce3cbc6a780" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 32861 } } }
I20260619 08:19:47.437147 15010 sys_catalog.cc:458] T 00000000000000000000000000000000 P df1e09cccc164c8f8418cce3cbc6a780 [sys.catalog]: This master's current role is: LEADER
I20260619 08:19:47.437885 15013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:19:47.438485 15013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:19:47.439939 15013 catalog_manager.cc:1347] Generated new cluster ID: 2e8736322f6f4aa3b66307d7eec07e14
I20260619 08:19:47.439990 15013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:19:47 dist-test-slave-d3n8 krb5kdc[14924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:47.449826 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:47.437538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52408 (local address 127.4.151.190:32861)
0619 08:19:47.438145 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:47.438154 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:47.438189 (+ 35us) server_negotiation.cc:407] Connection header received
0619 08:19:47.438317 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:47.438324 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:47.438381 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:47.438544 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:47.439341 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.440696 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.441774 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.442424 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.442538 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:47.445492 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:47.445496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:47.445497 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:47.445510 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:47.447955 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:47.448406 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:47.448410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:47.448410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:47.448522 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:47.448741 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:47.448742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:47.448742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:47.449058 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:47.449209 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:47.449324 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 08:19:47.449584 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":89,"threads_started":1}
I20260619 08:19:47.452559 15013 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:19:47.453075 15013 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:19:47.461004 15013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df1e09cccc164c8f8418cce3cbc6a780: Generated new TSK 0
I20260619 08:19:47.461215 15013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.151.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
I20260619 08:19:47.492815 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:32861
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:44459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:47.578390 15031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:47.578598 15031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:47.578621 15031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:47.579766 15031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:47.579809 15031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:47.579835 15031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:19:47.581277 15031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:44459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:32861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:47.581549 15031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:47.581805 15031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:47.587107 15039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:47.587174 15037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:47.587138 15036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:47.587188 15031 server_base.cc:1034] running on GCE node
I20260619 08:19:47.587493 15031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:47.587744 15031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:47.588910 15031 hybrid_clock.cc:648] HybridClock initialized: now 1781857187588880 us; error 50 us; skew 500 ppm
Jun 19 08:19:47 dist-test-slave-d3n8 krb5kdc[14924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857187, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:47.591600 15031 init.cc:377] Logged in from keytab as kudu/127.4.151.129@KRBTEST.COM (short username kudu)
I20260619 08:19:47.592147 15031 webserver.cc:460] Webserver started at http://127.4.151.129:41413/ using document root <none> and password file <none>
I20260619 08:19:47.592383 15031 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:47.592453 15031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:47.592605 15031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:47.593693 15031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "33d633cfa8454e60bf5275f4d9ecd47f"
format_stamp: "Formatted at 2026-06-19 08:19:47 on dist-test-slave-d3n8"
server_key: "de0287e07272aabf1a3bf3de7df3efaf"
server_key_iv: "a319b5289d3fe37ada739d06e55db770"
server_key_version: "encryptionkey@0"
I20260619 08:19:47.594065 15031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "33d633cfa8454e60bf5275f4d9ecd47f"
format_stamp: "Formatted at 2026-06-19 08:19:47 on dist-test-slave-d3n8"
server_key: "de0287e07272aabf1a3bf3de7df3efaf"
server_key_iv: "a319b5289d3fe37ada739d06e55db770"
server_key_version: "encryptionkey@0"
I20260619 08:19:47.595413 15031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:19:47.596194 15046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:47.596388 15031 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:47.596485 15031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "33d633cfa8454e60bf5275f4d9ecd47f"
format_stamp: "Formatted at 2026-06-19 08:19:47 on dist-test-slave-d3n8"
server_key: "de0287e07272aabf1a3bf3de7df3efaf"
server_key_iv: "a319b5289d3fe37ada739d06e55db770"
server_key_version: "encryptionkey@0"
I20260619 08:19:47.596566 15031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:47.605443 15031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:47.607664 15031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:47.607831 15031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:47.608070 15031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:47.608398 15031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:47.608454 15031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:47.608498 15031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:47.608521 15031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:47.614903 15031 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:45517
I20260619 08:19:47.614953 15159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:45517 every 8 connection(s)
I20260619 08:19:47.615296 15031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
I20260619 08:19:47.617689 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 15031
I20260619 08:19:47.617774 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:19:47.617983 4702 external_mini_cluster.cc:1428] Setting key f428adca585880953011d9f457d9c585
Jun 19 08:19:47 dist-test-slave-d3n8 krb5kdc[14924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857187, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:47.626056 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:47.616103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:47131 (local address 127.4.151.190:32861)
0619 08:19:47.616177 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:47.616179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:47.616330 (+ 151us) server_negotiation.cc:407] Connection header received
0619 08:19:47.616850 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:47.616851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:47.616862 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:47.616920 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:47.618047 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.618653 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.619783 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.620115 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.620191 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:47.622675 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:47.622677 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:47.622677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:47.622686 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:47.624418 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:47.624868 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:47.624869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:47.624870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:47.624933 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:47.625207 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:47.625208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:47.625209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:47.625354 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:47.625460 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:47.625851 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 08:19:47.625948 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 08:19:47.626169 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:47.616035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:32861 (local address 127.4.151.129:47131)
0619 08:19:47.616194 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:47.616230 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:47.616733 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:47.616972 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:47.616975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:47.617682 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:47.617938 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:47.617943 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.618755 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.618756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:47.619687 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:47.619689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.620240 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.620241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:47.620368 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:47.621028 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:47.621037 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:47.622581 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:47.624546 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:47.624552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:47.624554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:47.624783 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:47.625020 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:47.625021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:47.625021 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:47.625134 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:47.625579 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:47.625585 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:47.625705 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 08:19:47.625794 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 08:19:47.625921 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.4.151.130@KRBTEST.COM; defaulting to no policy
I20260619 08:19:47.626749 15160 heartbeater.cc:347] Connected to a master server at 127.4.151.190:32861
I20260619 08:19:47.626853 15160 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:47.627028 15160 heartbeater.cc:511] Master 127.4.151.190:32861 requested a full tablet report, sending...
I20260619 08:19:47.627548 14969 ts_manager.cc:194] Registered new tserver with Master: 33d633cfa8454e60bf5275f4d9ecd47f (127.4.151.129:45517)
I20260619 08:19:47.628190 14969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.129@KRBTEST.COM'} at 127.4.151.129:47131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
I20260619 08:19:47.644706 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:32861
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:44459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:47.726554 15165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:47.726807 15165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:47.726846 15165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:47.728030 15165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:47.728112 15165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:47.728158 15165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:19:47.729698 15165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:44459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:32861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:47.729977 15165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:47.730211 15165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:47.735718 15171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:47.735782 15173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:47.735718 15170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:47.735803 15165 server_base.cc:1034] running on GCE node
I20260619 08:19:47.736171 15165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:47.736418 15165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:47.737664 15165 hybrid_clock.cc:648] HybridClock initialized: now 1781857187737628 us; error 51 us; skew 500 ppm
Jun 19 08:19:47 dist-test-slave-d3n8 krb5kdc[14924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857187, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:47.740432 15165 init.cc:377] Logged in from keytab as kudu/127.4.151.130@KRBTEST.COM (short username kudu)
I20260619 08:19:47.740938 15165 webserver.cc:460] Webserver started at http://127.4.151.130:33551/ using document root <none> and password file <none>
I20260619 08:19:47.741209 15165 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:47.741285 15165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:47.741448 15165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:47.742548 15165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "43dbd87b33fe450ca92849eda8c96745"
format_stamp: "Formatted at 2026-06-19 08:19:47 on dist-test-slave-d3n8"
server_key: "2e7b9884c65c047987dcd040ed601df3"
server_key_iv: "9eaa18c3627c10dbde94452e45635572"
server_key_version: "encryptionkey@0"
I20260619 08:19:47.742931 15165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "43dbd87b33fe450ca92849eda8c96745"
format_stamp: "Formatted at 2026-06-19 08:19:47 on dist-test-slave-d3n8"
server_key: "2e7b9884c65c047987dcd040ed601df3"
server_key_iv: "9eaa18c3627c10dbde94452e45635572"
server_key_version: "encryptionkey@0"
I20260619 08:19:47.744274 15165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:19:47.745122 15180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:47.745352 15165 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:47.745486 15165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "43dbd87b33fe450ca92849eda8c96745"
format_stamp: "Formatted at 2026-06-19 08:19:47 on dist-test-slave-d3n8"
server_key: "2e7b9884c65c047987dcd040ed601df3"
server_key_iv: "9eaa18c3627c10dbde94452e45635572"
server_key_version: "encryptionkey@0"
I20260619 08:19:47.745608 15165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:47.757572 15165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:47.759510 15165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:47.759691 15165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:47.759930 15165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:47.760267 15165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:47.760324 15165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:47.760365 15165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:47.760394 15165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:47.766366 15165 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:35931
I20260619 08:19:47.766404 15293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:35931 every 8 connection(s)
I20260619 08:19:47.766824 15165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
I20260619 08:19:47.769672 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 15165
I20260619 08:19:47.769754 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:19:47.769934 4702 external_mini_cluster.cc:1428] Setting key 0451b2aeec762e53adf6fa6ac74a37d9
Jun 19 08:19:47 dist-test-slave-d3n8 krb5kdc[14924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857187, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:47.777784 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:47.767591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:58797 (local address 127.4.151.190:32861)
0619 08:19:47.767674 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:47.767676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:47.767845 (+ 169us) server_negotiation.cc:407] Connection header received
0619 08:19:47.768352 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:47.768354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:47.768365 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:47.768427 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:47.769359 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.770074 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.771369 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.771582 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.771640 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:47.774526 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:47.774527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:47.774527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:47.774534 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:47.776060 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:47.776617 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:47.776618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:47.776619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:47.776677 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:47.777019 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:47.777020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:47.777020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:47.777167 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:47.777231 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:47.777635 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 08:19:47.777716 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 08:19:47.778030 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:47.767556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:32861 (local address 127.4.151.130:58797)
0619 08:19:47.767727 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:47.767762 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:47.768236 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:47.768497 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:47.768500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:47.769086 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:47.769272 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:47.769277 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.770201 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.770202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:47.771281 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:47.771284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.771668 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.771669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:47.771817 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:47.772428 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:47.772438 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:47.774405 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:47.776155 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:47.776163 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:47.776166 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:47.776512 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:47.776757 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:47.776758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:47.776758 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:47.776899 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:47.777245 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:47.777251 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:47.777453 (+ 202us) client_negotiation.cc:769] Sending connection context
0619 08:19:47.777579 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 08:19:47.777768 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":100,"threads_started":1}
I20260619 08:19:47.778614 15294 heartbeater.cc:347] Connected to a master server at 127.4.151.190:32861
I20260619 08:19:47.778708 15294 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:47.778893 15294 heartbeater.cc:511] Master 127.4.151.190:32861 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.151.131@KRBTEST.COM; defaulting to no policy
I20260619 08:19:47.779284 14969 ts_manager.cc:194] Registered new tserver with Master: 43dbd87b33fe450ca92849eda8c96745 (127.4.151.130:35931)
I20260619 08:19:47.779628 14969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.130@KRBTEST.COM'} at 127.4.151.130:58797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
I20260619 08:19:47.798779 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:32861
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:44459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:19:47.883209 15299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:19:47.883466 15299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:19:47.883504 15299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:19:47.884670 15299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:19:47.884740 15299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:19:47.884786 15299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:19:47.886276 15299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:44459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:32861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:19:47.886545 15299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:19:47.886780 15299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:19:47.892184 15305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:47.892221 15299 server_base.cc:1034] running on GCE node
W20260619 08:19:47.892215 15307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:19:47.892166 15304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:19:47.892628 15299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:19:47.892956 15299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:19:47.894121 15299 hybrid_clock.cc:648] HybridClock initialized: now 1781857187894083 us; error 88 us; skew 500 ppm
Jun 19 08:19:47 dist-test-slave-d3n8 krb5kdc[14924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857187, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:19:47.896839 15299 init.cc:377] Logged in from keytab as kudu/127.4.151.131@KRBTEST.COM (short username kudu)
I20260619 08:19:47.897356 15299 webserver.cc:460] Webserver started at http://127.4.151.131:34019/ using document root <none> and password file <none>
I20260619 08:19:47.897770 15299 fs_manager.cc:359] Metadata directory not provided
I20260619 08:19:47.897843 15299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:19:47.897992 15299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:19:47.899101 15299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "1731695e29f3478cb1e0b0ff15f53b67"
format_stamp: "Formatted at 2026-06-19 08:19:47 on dist-test-slave-d3n8"
server_key: "5619f4082c5d336a97409d43b279ed91"
server_key_iv: "0ea6ac4c012b907b3783ae65a43da16e"
server_key_version: "encryptionkey@0"
I20260619 08:19:47.899571 15299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "1731695e29f3478cb1e0b0ff15f53b67"
format_stamp: "Formatted at 2026-06-19 08:19:47 on dist-test-slave-d3n8"
server_key: "5619f4082c5d336a97409d43b279ed91"
server_key_iv: "0ea6ac4c012b907b3783ae65a43da16e"
server_key_version: "encryptionkey@0"
I20260619 08:19:47.901283 15299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:19:47.902401 15314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:47.902621 15299 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:19:47.902737 15299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "1731695e29f3478cb1e0b0ff15f53b67"
format_stamp: "Formatted at 2026-06-19 08:19:47 on dist-test-slave-d3n8"
server_key: "5619f4082c5d336a97409d43b279ed91"
server_key_iv: "0ea6ac4c012b907b3783ae65a43da16e"
server_key_version: "encryptionkey@0"
I20260619 08:19:47.902817 15299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:19:47.910018 15299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:19:47.912343 15299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:19:47.912569 15299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:19:47.912844 15299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:19:47.913187 15299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:19:47.913247 15299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:47.913293 15299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:19:47.913328 15299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:19:47.919806 15299 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:37463
I20260619 08:19:47.919876 15427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:37463 every 8 connection(s)
I20260619 08:19:47.920259 15299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
I20260619 08:19:47.924422 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 15299
I20260619 08:19:47.924528 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:19:47.924726 4702 external_mini_cluster.cc:1428] Setting key 7c33de2206771940bd6ab7699853c7bb
Jun 19 08:19:47 dist-test-slave-d3n8 krb5kdc[14924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857187, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:19:47.930086 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:47.921136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:42767 (local address 127.4.151.190:32861)
0619 08:19:47.921189 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:47.921191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:47.921325 (+ 134us) server_negotiation.cc:407] Connection header received
0619 08:19:47.921902 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:47.921904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:47.921915 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:47.921983 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:47.922943 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.923486 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.924672 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.924971 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.925078 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:47.927380 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:47.927381 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:47.927381 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:47.927388 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:47.928682 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:47.929113 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:47.929114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:47.929115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:47.929169 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:47.929450 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:47.929450 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:47.929451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:47.929580 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:47.929635 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:47.929941 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 08:19:47.930024 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 08:19:47.930119 15430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:47.920982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:32861 (local address 127.4.151.131:42767)
0619 08:19:47.921175 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:47.921218 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:47.921776 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:47.922057 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:47.922061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:47.922638 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:19:47.922826 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:47.922830 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.923585 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.923586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:47.924581 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:47.924585 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.925089 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.925091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:47.925225 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:47.925820 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:19:47.925828 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:19:47.927278 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:19:47.928767 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:47.928772 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:47.928774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:47.929028 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:47.929237 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:19:47.929238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:19:47.929238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:19:47.929336 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:19:47.929640 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:19:47.929645 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:19:47.929759 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 08:19:47.929838 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 08:19:47.929958 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":126,"threads_started":1}
I20260619 08:19:47.930611 15428 heartbeater.cc:347] Connected to a master server at 127.4.151.190:32861
I20260619 08:19:47.930692 15428 heartbeater.cc:464] Registering TS with master...
I20260619 08:19:47.930825 15428 heartbeater.cc:511] Master 127.4.151.190:32861 requested a full tablet report, sending...
I20260619 08:19:47.931229 14969 ts_manager.cc:194] Registered new tserver with Master: 1731695e29f3478cb1e0b0ff15f53b67 (127.4.151.131:37463)
I20260619 08:19:47.931612 14969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.131@KRBTEST.COM'} at 127.4.151.131:42767
I20260619 08:19:47.936755 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:19:47.944818 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:47.938444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52418 (local address 127.4.151.190:32861)
0619 08:19:47.938495 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:47.938497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:47.938506 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:19:47.938576 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:47.938576 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:47.938587 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:47.938638 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:19:47.939237 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.939729 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.940479 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.940680 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.940766 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:47.942120 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:19:47.942121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:19:47.942122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:19:47.942130 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:19:47.943576 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:47.943957 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:47.943959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:47.943959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:47.944018 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:19:47.944225 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:19:47.944226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:19:47.944226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:19:47.944387 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:19:47.944497 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:47.944633 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 08:19:47.944714 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 08:19:47.946762 14969 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:19:47.948437 14969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:19:47.955189 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:47.951004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:35931 (local address 127.0.0.1:50806)
0619 08:19:47.951260 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:47.951271 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:47.951320 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:47.951526 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:47.951529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:47.951532 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:47.951601 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:47.951602 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.952955 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.952956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:47.954008 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:47.954011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.954873 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.954874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:47.954962 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:47.954981 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 08:19:47.955045 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 08:19:47.955100 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":57,"threads_started":1}
I20260619 08:19:47.955288 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:47.951270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42018 (local address 127.4.151.131:37463)
0619 08:19:47.951774 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:47.951778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:47.951786 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:19:47.951826 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:47.951828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:47.951839 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:47.951925 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:47.952037 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.952736 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.954390 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.954918 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.954990 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:47.955045 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:47.955164 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 08:19:47.955205 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":75,"threads_started":1}
I20260619 08:19:47.955331 15438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:47.951176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:37463 (local address 127.0.0.1:42018)
0619 08:19:47.951402 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:47.951411 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:47.951448 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:47.951923 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:47.951924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:47.951926 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:47.951981 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:47.951983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.952842 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.952843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:47.954284 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:47.954286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.954994 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.954995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:47.955075 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:47.955079 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:19:47.955139 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 08:19:47.955185 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":67,"thread_start_us":37,"threads_started":1}
I20260619 08:19:47.955649 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:47.950902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56394 (local address 127.4.151.129:45517)
0619 08:19:47.951112 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:47.951117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:47.951129 (+ 12us) server_negotiation.cc:407] Connection header received
0619 08:19:47.951261 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:47.951264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:47.951295 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:47.951384 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:47.951711 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.952813 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.954376 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.955084 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.955442 (+ 358us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:47.955487 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:47.955509 (+ 22us) server_negotiation.cc:299] Negotiation successful
0619 08:19:47.955541 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":95,"threads_started":1}
I20260619 08:19:47.955390 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:47.950787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:45517 (local address 127.0.0.1:56394)
0619 08:19:47.951018 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:47.951066 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:47.951144 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:47.951579 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:47.951580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:47.951591 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:47.951640 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:47.951642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.952940 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.952940 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:47.954295 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:47.954296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.955166 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.955167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:47.955222 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:47.955225 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:19:47.955260 (+ 35us) client_negotiation.cc:240] Negotiation successful
0619 08:19:47.955292 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":93,"thread_start_us":75,"threads_started":1}
I20260619 08:19:47.955962 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:47.951101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50806 (local address 127.4.151.130:35931)
0619 08:19:47.951290 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:47.951295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:47.951308 (+ 13us) server_negotiation.cc:407] Connection header received
0619 08:19:47.951398 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:47.951400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:47.951417 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:47.951495 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:47.951687 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.952847 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.954105 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:47.954781 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:47.954858 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:47.954924 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:47.955792 (+ 868us) server_negotiation.cc:299] Negotiation successful
0619 08:19:47.955855 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":76,"threads_started":1}
I20260619 08:19:47.956537 15362 tablet_service.cc:1467] Processing CreateTablet for tablet 5770fcf5bb7a476abda4f9a5d1bd1c4e (DEFAULT_TABLE table=test-table [id=ea7de3107e4746e0bb43b74fa257a957]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:47.956553 15094 tablet_service.cc:1467] Processing CreateTablet for tablet 5770fcf5bb7a476abda4f9a5d1bd1c4e (DEFAULT_TABLE table=test-table [id=ea7de3107e4746e0bb43b74fa257a957]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:47.956553 15228 tablet_service.cc:1467] Processing CreateTablet for tablet 5770fcf5bb7a476abda4f9a5d1bd1c4e (DEFAULT_TABLE table=test-table [id=ea7de3107e4746e0bb43b74fa257a957]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:19:47.956876 15362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5770fcf5bb7a476abda4f9a5d1bd1c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:47.956875 15228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5770fcf5bb7a476abda4f9a5d1bd1c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:47.956874 15094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5770fcf5bb7a476abda4f9a5d1bd1c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:19:47.959637 15442 tablet_bootstrap.cc:492] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 1731695e29f3478cb1e0b0ff15f53b67: Bootstrap starting.
I20260619 08:19:47.960161 15444 tablet_bootstrap.cc:492] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745: Bootstrap starting.
I20260619 08:19:47.960350 15442 tablet_bootstrap.cc:654] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 1731695e29f3478cb1e0b0ff15f53b67: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:47.960850 15444 tablet_bootstrap.cc:654] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:47.961035 15442 log.cc:824] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 1731695e29f3478cb1e0b0ff15f53b67: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:47.961516 15444 log.cc:824] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:47.961571 15443 tablet_bootstrap.cc:492] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 33d633cfa8454e60bf5275f4d9ecd47f: Bootstrap starting.
I20260619 08:19:47.962175 15442 tablet_bootstrap.cc:492] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 1731695e29f3478cb1e0b0ff15f53b67: No bootstrap required, opened a new log
I20260619 08:19:47.962220 15443 tablet_bootstrap.cc:654] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 33d633cfa8454e60bf5275f4d9ecd47f: Neither blocks nor log segments found. Creating new log.
I20260619 08:19:47.962255 15442 ts_tablet_manager.cc:1397] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 1731695e29f3478cb1e0b0ff15f53b67: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 08:19:47.962427 15444 tablet_bootstrap.cc:492] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745: No bootstrap required, opened a new log
I20260619 08:19:47.962502 15444 ts_tablet_manager.cc:1397] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:19:47.962778 15443 log.cc:824] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 33d633cfa8454e60bf5275f4d9ecd47f: Log is configured to *not* fsync() on all Append() calls
I20260619 08:19:47.963552 15443 tablet_bootstrap.cc:492] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 33d633cfa8454e60bf5275f4d9ecd47f: No bootstrap required, opened a new log
I20260619 08:19:47.963622 15443 ts_tablet_manager.cc:1397] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 33d633cfa8454e60bf5275f4d9ecd47f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:19:47.964210 15444 raft_consensus.cc:348] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d633cfa8454e60bf5275f4d9ecd47f" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 45517 } } peers { permanent_uuid: "43dbd87b33fe450ca92849eda8c96745" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35931 } } peers { permanent_uuid: "1731695e29f3478cb1e0b0ff15f53b67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 37463 } }
I20260619 08:19:47.964363 15444 raft_consensus.cc:374] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:47.964408 15444 raft_consensus.cc:729] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43dbd87b33fe450ca92849eda8c96745, State: Initialized, Role: FOLLOWER
I20260619 08:19:47.964346 15442 raft_consensus.cc:348] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 1731695e29f3478cb1e0b0ff15f53b67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d633cfa8454e60bf5275f4d9ecd47f" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 45517 } } peers { permanent_uuid: "43dbd87b33fe450ca92849eda8c96745" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35931 } } peers { permanent_uuid: "1731695e29f3478cb1e0b0ff15f53b67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 37463 } }
I20260619 08:19:47.964506 15442 raft_consensus.cc:374] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 1731695e29f3478cb1e0b0ff15f53b67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:47.964542 15442 raft_consensus.cc:729] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 1731695e29f3478cb1e0b0ff15f53b67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1731695e29f3478cb1e0b0ff15f53b67, State: Initialized, Role: FOLLOWER
I20260619 08:19:47.964548 15444 consensus_queue.cc:260] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d633cfa8454e60bf5275f4d9ecd47f" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 45517 } } peers { permanent_uuid: "43dbd87b33fe450ca92849eda8c96745" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35931 } } peers { permanent_uuid: "1731695e29f3478cb1e0b0ff15f53b67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 37463 } }
I20260619 08:19:47.964686 15442 consensus_queue.cc:260] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 1731695e29f3478cb1e0b0ff15f53b67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d633cfa8454e60bf5275f4d9ecd47f" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 45517 } } peers { permanent_uuid: "43dbd87b33fe450ca92849eda8c96745" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35931 } } peers { permanent_uuid: "1731695e29f3478cb1e0b0ff15f53b67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 37463 } }
I20260619 08:19:47.964893 15294 heartbeater.cc:503] Master 127.4.151.190:32861 was elected leader, sending a full tablet report...
I20260619 08:19:47.964962 15442 ts_tablet_manager.cc:1428] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 1731695e29f3478cb1e0b0ff15f53b67: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:19:47.965082 15428 heartbeater.cc:503] Master 127.4.151.190:32861 was elected leader, sending a full tablet report...
I20260619 08:19:47.964840 15444 ts_tablet_manager.cc:1428] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:19:47.965986 15443 raft_consensus.cc:348] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 33d633cfa8454e60bf5275f4d9ecd47f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d633cfa8454e60bf5275f4d9ecd47f" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 45517 } } peers { permanent_uuid: "43dbd87b33fe450ca92849eda8c96745" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35931 } } peers { permanent_uuid: "1731695e29f3478cb1e0b0ff15f53b67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 37463 } }
I20260619 08:19:47.966125 15443 raft_consensus.cc:374] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 33d633cfa8454e60bf5275f4d9ecd47f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:19:47.966168 15443 raft_consensus.cc:729] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 33d633cfa8454e60bf5275f4d9ecd47f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d633cfa8454e60bf5275f4d9ecd47f, State: Initialized, Role: FOLLOWER
I20260619 08:19:47.966274 15443 consensus_queue.cc:260] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 33d633cfa8454e60bf5275f4d9ecd47f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d633cfa8454e60bf5275f4d9ecd47f" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 45517 } } peers { permanent_uuid: "43dbd87b33fe450ca92849eda8c96745" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35931 } } peers { permanent_uuid: "1731695e29f3478cb1e0b0ff15f53b67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 37463 } }
I20260619 08:19:47.966538 15443 ts_tablet_manager.cc:1428] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 33d633cfa8454e60bf5275f4d9ecd47f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:19:47.966559 15160 heartbeater.cc:503] Master 127.4.151.190:32861 was elected leader, sending a full tablet report...
W20260619 08:19:48.017457 15295 tablet.cc:2369] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:19:48.059195 15448 raft_consensus.cc:482] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:19:48.059342 15448 raft_consensus.cc:504] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d633cfa8454e60bf5275f4d9ecd47f" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 45517 } } peers { permanent_uuid: "43dbd87b33fe450ca92849eda8c96745" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35931 } } peers { permanent_uuid: "1731695e29f3478cb1e0b0ff15f53b67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 37463 } }
I20260619 08:19:48.059748 15448 leader_election.cc:288] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33d633cfa8454e60bf5275f4d9ecd47f (127.4.151.129:45517), 1731695e29f3478cb1e0b0ff15f53b67 (127.4.151.131:37463)
I20260619 08:19:48.062958 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:48.060269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:36041 (local address 127.4.151.131:37463)
0619 08:19:48.060338 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:48.060340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:48.060452 (+ 112us) server_negotiation.cc:407] Connection header received
0619 08:19:48.060527 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:48.060528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:48.060537 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:48.060615 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:48.060770 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:48.061258 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:48.062219 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:48.062602 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:48.062692 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:48.062700 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:48.062845 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 08:19:48.062881 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 08:19:48.063017 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:48.060111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:37335 (local address 127.4.151.129:45517)
0619 08:19:48.060198 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:48.060201 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:48.060214 (+ 13us) server_negotiation.cc:407] Connection header received
0619 08:19:48.060248 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:48.060249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:48.060263 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:48.060332 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:19:48.060525 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:48.061088 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:48.062168 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:48.062587 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:48.062708 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:48.062716 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:48.062889 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 08:19:48.062930 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 08:19:48.063017 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:48.059969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:45517 (local address 127.4.151.130:37335)
0619 08:19:48.060098 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:48.060113 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:48.060182 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:48.060351 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:48.060352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:48.060354 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:48.060432 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:48.060434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:48.061188 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:48.061189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:48.062050 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:48.062052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:48.062709 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:48.062710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:48.062803 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:48.062805 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:19:48.062882 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 08:19:48.062927 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 08:19:48.063024 15451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:48.060162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:37463 (local address 127.4.151.130:36041)
0619 08:19:48.060375 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:48.060391 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 08:19:48.060458 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:19:48.060619 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:19:48.060620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:19:48.060622 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:19:48.060698 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:48.060699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:48.061347 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:48.061348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:48.062124 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:19:48.062126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:19:48.062692 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:19:48.062693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:19:48.062769 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:48.062773 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:19:48.062841 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 08:19:48.062878 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":126,"threads_started":1}
I20260619 08:19:48.063405 15114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5770fcf5bb7a476abda4f9a5d1bd1c4e" candidate_uuid: "43dbd87b33fe450ca92849eda8c96745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d633cfa8454e60bf5275f4d9ecd47f" is_pre_election: true
I20260619 08:19:48.063410 15382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5770fcf5bb7a476abda4f9a5d1bd1c4e" candidate_uuid: "43dbd87b33fe450ca92849eda8c96745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1731695e29f3478cb1e0b0ff15f53b67" is_pre_election: true
I20260619 08:19:48.063571 15382 raft_consensus.cc:2454] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 1731695e29f3478cb1e0b0ff15f53b67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43dbd87b33fe450ca92849eda8c96745 in term 0.
I20260619 08:19:48.063581 15114 raft_consensus.cc:2454] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 33d633cfa8454e60bf5275f4d9ecd47f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43dbd87b33fe450ca92849eda8c96745 in term 0.
I20260619 08:19:48.063827 15181 leader_election.cc:302] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1731695e29f3478cb1e0b0ff15f53b67, 43dbd87b33fe450ca92849eda8c96745; no voters:
I20260619 08:19:48.063988 15448 raft_consensus.cc:2781] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:19:48.064143 15448 raft_consensus.cc:482] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:19:48.064163 15448 raft_consensus.cc:3037] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:48.064932 15448 raft_consensus.cc:504] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d633cfa8454e60bf5275f4d9ecd47f" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 45517 } } peers { permanent_uuid: "43dbd87b33fe450ca92849eda8c96745" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35931 } } peers { permanent_uuid: "1731695e29f3478cb1e0b0ff15f53b67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 37463 } }
I20260619 08:19:48.065057 15448 leader_election.cc:288] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745 [CANDIDATE]: Term 1 election: Requested vote from peers 33d633cfa8454e60bf5275f4d9ecd47f (127.4.151.129:45517), 1731695e29f3478cb1e0b0ff15f53b67 (127.4.151.131:37463)
I20260619 08:19:48.065310 15114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5770fcf5bb7a476abda4f9a5d1bd1c4e" candidate_uuid: "43dbd87b33fe450ca92849eda8c96745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d633cfa8454e60bf5275f4d9ecd47f"
I20260619 08:19:48.065361 15382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5770fcf5bb7a476abda4f9a5d1bd1c4e" candidate_uuid: "43dbd87b33fe450ca92849eda8c96745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1731695e29f3478cb1e0b0ff15f53b67"
I20260619 08:19:48.065435 15114 raft_consensus.cc:3037] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 33d633cfa8454e60bf5275f4d9ecd47f [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:48.065495 15382 raft_consensus.cc:3037] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 1731695e29f3478cb1e0b0ff15f53b67 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:19:48.066253 15114 raft_consensus.cc:2454] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 33d633cfa8454e60bf5275f4d9ecd47f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43dbd87b33fe450ca92849eda8c96745 in term 1.
I20260619 08:19:48.066462 15181 leader_election.cc:302] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d633cfa8454e60bf5275f4d9ecd47f, 43dbd87b33fe450ca92849eda8c96745; no voters:
I20260619 08:19:48.066521 15382 raft_consensus.cc:2454] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 1731695e29f3478cb1e0b0ff15f53b67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43dbd87b33fe450ca92849eda8c96745 in term 1.
I20260619 08:19:48.066583 15448 raft_consensus.cc:2781] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:19:48.066756 15448 raft_consensus.cc:686] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745 [term 1 LEADER]: Becoming Leader. State: Replica: 43dbd87b33fe450ca92849eda8c96745, State: Running, Role: LEADER
I20260619 08:19:48.066891 15448 consensus_queue.cc:237] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d633cfa8454e60bf5275f4d9ecd47f" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 45517 } } peers { permanent_uuid: "43dbd87b33fe450ca92849eda8c96745" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35931 } } peers { permanent_uuid: "1731695e29f3478cb1e0b0ff15f53b67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 37463 } }
I20260619 08:19:48.067813 14968 catalog_manager.cc:5495] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43dbd87b33fe450ca92849eda8c96745 (127.4.151.130). New cstate: current_term: 1 leader_uuid: "43dbd87b33fe450ca92849eda8c96745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d633cfa8454e60bf5275f4d9ecd47f" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 45517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43dbd87b33fe450ca92849eda8c96745" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 35931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1731695e29f3478cb1e0b0ff15f53b67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 37463 } health_report { overall_health: UNKNOWN } } }
I20260619 08:19:48.100682 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:48.098142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50814 (local address 127.4.151.130:35931)
0619 08:19:48.098205 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 08:19:48.098207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:19:48.098216 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:19:48.098264 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:19:48.098265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:19:48.098282 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:19:48.098343 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:19:48.098476 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:48.098960 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:48.099895 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:19:48.100136 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:19:48.100195 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:19:48.100318 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:19:48.100411 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:19:48.100467 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 08:19:48.100525 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 08:19:48.100542 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 08:19:48.102885 15382 raft_consensus.cc:1261] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 1731695e29f3478cb1e0b0ff15f53b67 [term 1 FOLLOWER]: Refusing update from remote peer 43dbd87b33fe450ca92849eda8c96745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:19:48.102885 15114 raft_consensus.cc:1261] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 33d633cfa8454e60bf5275f4d9ecd47f [term 1 FOLLOWER]: Refusing update from remote peer 43dbd87b33fe450ca92849eda8c96745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:19:48.103276 15448 consensus_queue.cc:1035] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d633cfa8454e60bf5275f4d9ecd47f" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 45517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:19:48.103473 15452 consensus_queue.cc:1035] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1731695e29f3478cb1e0b0ff15f53b67" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 37463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:19:48.106621 15457 mvcc.cc:205] Tried to move back new op lower bound from 7298487042465554432 to 7298487042322677760. Current Snapshot: MvccSnapshot[applied={T|T < 7298487042465554432}]
I20260619 08:19:48.107934 14969 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:19:48.108038 14969 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:19:48.108767 14969 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df1e09cccc164c8f8418cce3cbc6a780: Sending DeleteTablet for 3 replicas of tablet 5770fcf5bb7a476abda4f9a5d1bd1c4e
I20260619 08:19:48.109138 15228 tablet_service.cc:1514] Processing DeleteTablet for tablet 5770fcf5bb7a476abda4f9a5d1bd1c4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:48 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:50806
I20260619 08:19:48.109164 15094 tablet_service.cc:1514] Processing DeleteTablet for tablet 5770fcf5bb7a476abda4f9a5d1bd1c4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:48 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:56394
I20260619 08:19:48.109375 15465 tablet_replica.cc:330] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 33d633cfa8454e60bf5275f4d9ecd47f: stopping tablet replica
I20260619 08:19:48.109441 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 15031
I20260619 08:19:48.109450 15362 tablet_service.cc:1514] Processing DeleteTablet for tablet 5770fcf5bb7a476abda4f9a5d1bd1c4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:19:48 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:42018
I20260619 08:19:48.109509 15465 raft_consensus.cc:2229] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 33d633cfa8454e60bf5275f4d9ecd47f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:19:48.109616 15465 raft_consensus.cc:2258] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 33d633cfa8454e60bf5275f4d9ecd47f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:48.109844 15466 tablet_replica.cc:330] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745: stopping tablet replica
I20260619 08:19:48.109974 15467 tablet_replica.cc:330] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 1731695e29f3478cb1e0b0ff15f53b67: stopping tablet replica
I20260619 08:19:48.110067 15467 raft_consensus.cc:2229] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 1731695e29f3478cb1e0b0ff15f53b67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:19:48.110162 15467 raft_consensus.cc:2258] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 1731695e29f3478cb1e0b0ff15f53b67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:48.110302 15466 raft_consensus.cc:2229] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:19:48.110445 15466 raft_consensus.cc:2258] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:19:48.110885 15466 ts_tablet_manager.cc:1905] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:48.111824 15467 ts_tablet_manager.cc:1905] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 1731695e29f3478cb1e0b0ff15f53b67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:19:48.112622 15466 ts_tablet_manager.cc:1918] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:19:48.112701 15466 log.cc:1196] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/5770fcf5bb7a476abda4f9a5d1bd1c4e
I20260619 08:19:48.112995 15466 ts_tablet_manager.cc:1939] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 43dbd87b33fe450ca92849eda8c96745: Deleting consensus metadata
I20260619 08:19:48.113416 14953 catalog_manager.cc:4841] TS 43dbd87b33fe450ca92849eda8c96745 (127.4.151.130:35931): tablet 5770fcf5bb7a476abda4f9a5d1bd1c4e (table test-table [id=ea7de3107e4746e0bb43b74fa257a957]) successfully deleted
I20260619 08:19:48.113765 15467 ts_tablet_manager.cc:1918] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 1731695e29f3478cb1e0b0ff15f53b67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:19:48.113855 15467 log.cc:1196] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 1731695e29f3478cb1e0b0ff15f53b67: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/5770fcf5bb7a476abda4f9a5d1bd1c4e
I20260619 08:19:48.114121 15467 ts_tablet_manager.cc:1939] T 5770fcf5bb7a476abda4f9a5d1bd1c4e P 1731695e29f3478cb1e0b0ff15f53b67: Deleting consensus metadata
I20260619 08:19:48.114610 14953 catalog_manager.cc:4841] TS 1731695e29f3478cb1e0b0ff15f53b67 (127.4.151.131:37463): tablet 5770fcf5bb7a476abda4f9a5d1bd1c4e (table test-table [id=ea7de3107e4746e0bb43b74fa257a957]) successfully deleted
W20260619 08:19:48.117660 14953 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.151.129:45517 (error 108)
I20260619 08:19:48.117928 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 15165
I20260619 08:19:48.118744 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:19:48.118111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:45517 (local address 127.0.0.1:56396)
0619 08:19:48.118681 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0619 08:19:48.118715 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.129:45517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550}
W20260619 08:19:48.118911 14953 catalog_manager.cc:4584] TS 33d633cfa8454e60bf5275f4d9ecd47f (127.4.151.129:45517): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5770fcf5bb7a476abda4f9a5d1bd1c4e: Network error: Client connection negotiation failed: client connection to 127.4.151.129:45517: connect: Connection refused (error 111)
I20260619 08:19:48.123188 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 15299
I20260619 08:19:48.127681 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 14937
2026-06-19T08:19:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2006 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:19:48 dist-test-slave-d3n8 krb5kdc[15471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:19:48 dist-test-slave-d3n8 krb5kdc[15471](info): set up 2 sockets
Jun 19 08:19:48 dist-test-slave-d3n8 krb5kdc[15471](info): commencing operation
krb5kdc: starting...
W20260619 08:19:49.174546 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:19:49 dist-test-slave-d3n8 krb5kdc[15471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:19:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:19:49Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.151.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.151.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.151.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/rangeradmin_127.4.151.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.151.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/rangeradmin_127.4.151.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.151.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.151.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.151.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/rangerlookup_127.4.151.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.151.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/rangerlookup_127.4.151.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.151.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.151.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.151.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/HTTP_127.4.151.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.151.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/HTTP_127.4.151.148@KRBTEST.COM.keytab.
I20260619 08:19:49.352753 4702 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskR4iS53/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/postgres -l logfile start
2026-06-19 08:19:52.269 UTC [15501] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 08:19:52.269 UTC [15501] LOG: listening on IPv4 address "127.4.151.148", port 37755
2026-06-19 08:19:52.274 UTC [15501] LOG: listening on Unix socket "/tmp/.s.PGSQL.37755"
2026-06-19 08:19:52.307 UTC [15504] LOG: database system was shut down at 2026-06-19 08:19:50 UTC
2026-06-19 08:19:52.311 UTC [15501] LOG: database system is ready to accept connections
I20260619 08:19:53.258481 4702 mini_postgres.cc:96] Postgres bound to 37755
2026-06-19 08:19:53.264 UTC [15512] FATAL: database "slave" does not exist
127.4.151.148:37755 - accepting connections
I20260619 08:19:53.265173 4702 mini_ranger.cc:161] Starting Ranger...
I20260619 08:19:53.279886 4702 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 08:19:53.830375 4702 mini_ranger.cc:87] Created ranger Postgres database
I20260619 08:19:53.830482 4702 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ranger-admin
2026-06-19 08:19:54,109 [I] DB FLAVOR :POSTGRES
2026-06-19 08:19:54,109 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 08:19:54,109 [I] Checking connection..
2026-06-19 08:19:54,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:37755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 08:19:54,353 [I] Checking connection passed.
2026-06-19 08:19:54,353 [I] --------- Verifying version history table ---------
2026-06-19 08:19:54,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:37755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 08:19:54,606 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 08:19:54,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:37755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 08:19:54,864 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 08:19:54,865 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 08:19:54,865 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:37755/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 08:19:55.102 UTC [15576] WARNING: there is no transaction in progress
2026-06-19 08:19:55,110 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 08:19:55,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:37755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 08:19:55,356 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 08:19:55,357 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 08:19:55,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:37755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 08:19:55,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:37755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 08:19:55,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:37755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-d3n8.c.gcp-upstream.internal','N') ;"
2026-06-19 08:19:56,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:37755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 08:19:56,356 [I] Table x_portal_user does not exist in database ranger
2026-06-19 08:19:56,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:37755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 08:19:56,599 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 08:19:56,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:37755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 08:19:56,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:37755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 08:19:57,142 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 08:19:57,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:37755/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 08:19:58.195 UTC [15684] WARNING: there is no transaction in progress
2026-06-19 08:19:58.213 UTC [15684] WARNING: there is no transaction in progress
2026-06-19 08:19:58.231 UTC [15684] WARNING: there is no transaction in progress
2026-06-19 08:19:58.250 UTC [15684] WARNING: there is no transaction in progress
2026-06-19 08:19:58.268 UTC [15684] WARNING: there is no transaction in progress
2026-06-19 08:19:58.286 UTC [15684] WARNING: there is no transaction in progress
2026-06-19 08:19:58.346 UTC [15684] WARNING: there is no transaction in progress
2026-06-19 08:19:58.354 UTC [15684] WARNING: there is no transaction in progress
2026-06-19 08:19:58.371 UTC [15684] WARNING: there is no transaction in progress
2026-06-19 08:19:58.381 UTC [15684] WARNING: there is no transaction in progress
2026-06-19 08:19:58.392 UTC [15684] WARNING: there is no transaction in progress
2026-06-19 08:19:58.408 UTC [15684] WARNING: there is no transaction in progress
2026-06-19 08:19:58.419 UTC [15684] WARNING: there is no transaction in progress
2026-06-19 08:19:58.426 UTC [15684] WARNING: there is no transaction in progress
2026-06-19 08:19:58.433 UTC [15684] WARNING: there is no transaction in progress
2026-06-19 08:19:58.441 UTC [15684] WARNING: there is no transaction in progress
2026-06-19 08:19:59,170 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 08:19:59,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:37755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 08:19:59,396 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 08:19:59,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:37755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 08:19:59,695 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 08:19:59,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:37755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 08:19:59,968 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 08:19:59,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:37755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 08:20:00,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:37755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 08:20:00,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:37755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-d3n8.c.gcp-upstream.internal';"
2026-06-19 08:20:00,739 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 08:20:00,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:37755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 08:20:00,999 [I] DB_PATCHES have already been applied
I20260619 08:20:01.005990 4702 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ranger-admin:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskR4iS53/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 08:20:01.006062 4702 mini_ranger.cc:193] Using host: 127.4.151.148
I20260619 08:20:01.009469 4702 mini_ranger.cc:227] Ranger admin URL: http://127.4.151.148:37233
Jun 19, 2026 8:20:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 8:20:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 8:20:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 8:20:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 8:20:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 8:20:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 8:20:01 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 08:20:02.217536 4702 mini_ranger.cc:160] Time spent starting Ranger: real 8.952s user 0.000s sys 0.010s
Jun 19, 2026 8:20:23 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 8:20:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 19, 2026 8:20:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 8:20:23 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 8:20:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 8:20:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 8:20:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 8:20:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 8:20:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 8:20:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 8:20:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 8:20:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 8:20:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 8:20:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 8:20:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 8:20:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 8:20:23 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 8:20:24 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 08:20:24.854058 4702 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.151.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.151.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.151.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/rangerkms_127.4.151.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.151.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/rangerkms_127.4.151.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.151.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.151.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.151.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/HTTP_127.4.151.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.151.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/HTTP_127.4.151.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 08:20:24 dist-test-slave-d3n8 krb5kdc[15471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857224, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 08:20:24.942821 4702 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 08:20:24.953276 4702 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 08:20:25.964479 4702 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 08:20:25.964569 4702 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ranger-kms
I20260619 08:20:25.964592 4702 mini_ranger_kms.cc:227] Using postgres at 127.4.151.148:37755
2026-06-19 08:20:26,144 [I] DB FLAVOR :POSTGRES
2026-06-19 08:20:26,144 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 08:20:26,144 [I] Checking connection
2026-06-19 08:20:26,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:37755/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 08:20:26,379 [I] connection success
2026-06-19 08:20:26,379 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 08:20:26,379 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 08:20:26,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:37755/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 08:20:26,632 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 08:20:26,633 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 08:20:26,633 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 08:20:26,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:37755/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 08:20:26,896 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 08:20:27.397161 4702 mini_ranger_kms.cc:310] Created kms service
I20260619 08:20:27.564323 4702 mini_ranger_kms.cc:326] Created kudu user
I20260619 08:20:27.607470 4702 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 08:20:27 dist-test-slave-d3n8 krb5kdc[15471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857224, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.151.148@KRBTEST.COM
I20260619 08:20:28.076315 4702 mini_ranger_kms.cc:384] Added ranger policy
I20260619 08:20:28.076431 4702 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ranger-kms:/tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 08:20:28.076449 4702 mini_ranger_kms.cc:240] Using host: 127.4.151.148
I20260619 08:20:28.079864 4702 mini_ranger_kms.cc:276] Ranger KMS PID: 15865
I20260619 08:20:28.079988 4702 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.151.148:47465
Jun 19, 2026 8:20:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 8:20:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 8:20:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 8:20:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 8:20:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 8:20:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 08:20:30.414778 4702 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.472s user 0.002s sys 0.161s
I20260619 08:20:30.414891 4702 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 08:20:30.414978 4702 mini_ranger_kms.cc:401] 127.4.151.148:47465/kms/v1/keys
Jun 19 08:20:30 dist-test-slave-d3n8 krb5kdc[15471](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781857230, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.151.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 08:20:30 dist-test-slave-d3n8 krb5kdc[15471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857230, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.151.148@KRBTEST.COM for HTTP/127.4.151.148@KRBTEST.COM
Jun 19 08:20:31 dist-test-slave-d3n8 krb5kdc[15471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857230, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.151.148@KRBTEST.COM for HTTP/127.4.151.148@KRBTEST.COM
Jun 19 08:20:33 dist-test-slave-d3n8 krb5kdc[15471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:20:33.580317 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:43947
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:43947
--ranger_config_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.151.148:47465/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:20:33.693066 15915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:20:33.693317 15915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:20:33.693356 15915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:20:33.694548 15915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:20:33.694615 15915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:20:33.694639 15915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:20:33.694667 15915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:20:33.694698 15915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:20:33.696290 15915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.151.148:47465/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:43947
--ranger_config_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:43947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:20:33.696568 15915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:20:33.696792 15915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:20:33.703290 15915 server_base.cc:1034] running on GCE node
W20260619 08:20:33.703332 15920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:20:33.703476 15921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:20:33.703461 15923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:20:33.703796 15915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:20:33.704137 15915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:20:33.705298 15915 hybrid_clock.cc:648] HybridClock initialized: now 1781857233705278 us; error 35 us; skew 500 ppm
Jun 19 08:20:33 dist-test-slave-d3n8 krb5kdc[15471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857233, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:20:33.708393 15915 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:20:33.708959 15915 webserver.cc:460] Webserver started at http://127.4.151.190:35541/ using document root <none> and password file <none>
I20260619 08:20:33.709288 15915 fs_manager.cc:359] Metadata directory not provided
I20260619 08:20:33.709359 15915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:20:33.709568 15915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 08:20:33 dist-test-slave-d3n8 krb5kdc[15471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857233, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for HTTP/127.4.151.148@KRBTEST.COM
I20260619 08:20:33.756112 15915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "86d37d9c85734c64bd6f156975586234"
format_stamp: "Formatted at 2026-06-19 08:20:33 on dist-test-slave-d3n8"
server_key: "b25aa2db514e9fe2a6f7b4d2f5586b9c"
server_key_iv: "b08db13bbf130446b88754348cc2b510"
server_key_version: "kuduclusterkey@0"
I20260619 08:20:33.756641 15915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "86d37d9c85734c64bd6f156975586234"
format_stamp: "Formatted at 2026-06-19 08:20:33 on dist-test-slave-d3n8"
server_key: "b25aa2db514e9fe2a6f7b4d2f5586b9c"
server_key_iv: "b08db13bbf130446b88754348cc2b510"
server_key_version: "kuduclusterkey@0"
I20260619 08:20:33.758381 15915 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 08:20:33.770231 15930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:20:33.770445 15915 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:20:33.770545 15915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "86d37d9c85734c64bd6f156975586234"
format_stamp: "Formatted at 2026-06-19 08:20:33 on dist-test-slave-d3n8"
server_key: "b25aa2db514e9fe2a6f7b4d2f5586b9c"
server_key_iv: "b08db13bbf130446b88754348cc2b510"
server_key_version: "kuduclusterkey@0"
I20260619 08:20:33.770666 15915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:20:33.789045 15915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:20:33.791079 15915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:20:33.791263 15915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:20:33.796236 15915 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:43947
I20260619 08:20:33.796260 15982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:43947 every 8 connection(s)
I20260619 08:20:33.796653 15915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:20:33.797050 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 15915
I20260619 08:20:33.797152 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:20:33.797348 4702 external_mini_cluster.cc:1428] Setting key 987088f17b64b5c88cdd9ef8df7241b6
I20260619 08:20:33.797550 15983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:20:33.801688 15983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86d37d9c85734c64bd6f156975586234: Bootstrap starting.
I20260619 08:20:33.802506 15983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86d37d9c85734c64bd6f156975586234: Neither blocks nor log segments found. Creating new log.
I20260619 08:20:33.802836 15983 log.cc:824] T 00000000000000000000000000000000 P 86d37d9c85734c64bd6f156975586234: Log is configured to *not* fsync() on all Append() calls
I20260619 08:20:33.803601 15983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86d37d9c85734c64bd6f156975586234: No bootstrap required, opened a new log
Jun 19 08:20:33 dist-test-slave-d3n8 krb5kdc[15471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:20:33.806516 15983 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86d37d9c85734c64bd6f156975586234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d37d9c85734c64bd6f156975586234" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43947 } }
I20260619 08:20:33.806737 15983 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86d37d9c85734c64bd6f156975586234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:20:33.806816 15983 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86d37d9c85734c64bd6f156975586234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86d37d9c85734c64bd6f156975586234, State: Initialized, Role: FOLLOWER
I20260619 08:20:33.806962 15983 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86d37d9c85734c64bd6f156975586234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d37d9c85734c64bd6f156975586234" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43947 } }
I20260619 08:20:33.807055 15983 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86d37d9c85734c64bd6f156975586234 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:20:33.807112 15983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86d37d9c85734c64bd6f156975586234 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:20:33.807171 15983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86d37d9c85734c64bd6f156975586234 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:20:33.807947 15983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86d37d9c85734c64bd6f156975586234 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d37d9c85734c64bd6f156975586234" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43947 } }
I20260619 08:20:33.808070 15983 leader_election.cc:302] T 00000000000000000000000000000000 P 86d37d9c85734c64bd6f156975586234 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86d37d9c85734c64bd6f156975586234; no voters:
I20260619 08:20:33.808265 15983 leader_election.cc:288] T 00000000000000000000000000000000 P 86d37d9c85734c64bd6f156975586234 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:20:33.808617 15989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86d37d9c85734c64bd6f156975586234 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:20:33.808746 15983 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86d37d9c85734c64bd6f156975586234 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:20:33.808846 15989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86d37d9c85734c64bd6f156975586234 [term 1 LEADER]: Becoming Leader. State: Replica: 86d37d9c85734c64bd6f156975586234, State: Running, Role: LEADER
I20260619 08:20:33.808991 15989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86d37d9c85734c64bd6f156975586234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d37d9c85734c64bd6f156975586234" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43947 } }
I20260619 08:20:33.809461 15989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86d37d9c85734c64bd6f156975586234 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86d37d9c85734c64bd6f156975586234" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d37d9c85734c64bd6f156975586234" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43947 } } }
I20260619 08:20:33.809790 15989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86d37d9c85734c64bd6f156975586234 [sys.catalog]: This master's current role is: LEADER
I20260619 08:20:33.809990 15989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86d37d9c85734c64bd6f156975586234 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86d37d9c85734c64bd6f156975586234. Latest consensus state: current_term: 1 leader_uuid: "86d37d9c85734c64bd6f156975586234" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d37d9c85734c64bd6f156975586234" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 43947 } } }
I20260619 08:20:33.810091 15989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86d37d9c85734c64bd6f156975586234 [sys.catalog]: This master's current role is: LEADER
I20260619 08:20:33.810338 15997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:20:33.811062 15997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:20:33.811440 15983 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 08:20:33.811499 15986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:20:33.798007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46062 (local address 127.4.151.190:43947)
0619 08:20:33.798202 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 08:20:33.798211 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:20:33.798339 (+ 128us) server_negotiation.cc:407] Connection header received
0619 08:20:33.799096 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:20:33.799107 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:20:33.799178 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:20:33.799482 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:20:33.800371 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:20:33.801834 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:20:33.803023 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:20:33.803681 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:20:33.803826 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:20:33.807138 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:20:33.807143 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:20:33.807144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:20:33.807156 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:20:33.809282 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:20:33.809766 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:20:33.809769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:20:33.809770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:20:33.809870 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:20:33.810169 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:20:33.810170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:20:33.810170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:20:33.810485 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:20:33.811071 (+ 586us) server_negotiation.cc:1035] Waiting for connection context
0619 08:20:33.811162 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 08:20:33.811285 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":81,"threads_started":1}
I20260619 08:20:33.822064 15997 catalog_manager.cc:1347] Generated new cluster ID: 422d4945f1454205b3f03de79488000b
I20260619 08:20:33.822235 15997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:20:33.831881 15997 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:20:33.832751 15997 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:20:33.848008 15997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86d37d9c85734c64bd6f156975586234: Generated new TSK 0
I20260619 08:20:33.848325 15997 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.151.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
I20260619 08:20:34.625908 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.151.148:47465/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43947
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:20:34.723580 16028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:20:34.723877 16028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:20:34.723909 16028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:20:34.725185 16028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:20:34.725231 16028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:20:34.725260 16028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:20:34.726907 16028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.151.148:47465/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:20:34.727154 16028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:20:34.727416 16028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:20:34.733256 16036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:20:34.733384 16034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:20:34.733256 16033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:20:34.733755 16028 server_base.cc:1034] running on GCE node
I20260619 08:20:34.734016 16028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:20:34.734229 16028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:20:34.735435 16028 hybrid_clock.cc:648] HybridClock initialized: now 1781857234735416 us; error 32 us; skew 500 ppm
Jun 19 08:20:34 dist-test-slave-d3n8 krb5kdc[15471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857234, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:20:34.738147 16028 init.cc:377] Logged in from keytab as kudu/127.4.151.129@KRBTEST.COM (short username kudu)
I20260619 08:20:34.738679 16028 webserver.cc:460] Webserver started at http://127.4.151.129:44149/ using document root <none> and password file <none>
I20260619 08:20:34.738936 16028 fs_manager.cc:359] Metadata directory not provided
I20260619 08:20:34.739005 16028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:20:34.739158 16028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 08:20:34 dist-test-slave-d3n8 krb5kdc[15471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857234, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for HTTP/127.4.151.148@KRBTEST.COM
I20260619 08:20:34.754400 16028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "352d9717c3d049128c48c41cdf7b5ba7"
format_stamp: "Formatted at 2026-06-19 08:20:34 on dist-test-slave-d3n8"
server_key: "ed8445f3a7fd5285610a759420d0ccce"
server_key_iv: "27f557bff19c829511912c4def934fc2"
server_key_version: "kuduclusterkey@0"
I20260619 08:20:34.754818 16028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "352d9717c3d049128c48c41cdf7b5ba7"
format_stamp: "Formatted at 2026-06-19 08:20:34 on dist-test-slave-d3n8"
server_key: "ed8445f3a7fd5285610a759420d0ccce"
server_key_iv: "27f557bff19c829511912c4def934fc2"
server_key_version: "kuduclusterkey@0"
I20260619 08:20:34.757625 16028 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 08:20:34.772040 16044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:20:34.772312 16028 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:20:34.772403 16028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "352d9717c3d049128c48c41cdf7b5ba7"
format_stamp: "Formatted at 2026-06-19 08:20:34 on dist-test-slave-d3n8"
server_key: "ed8445f3a7fd5285610a759420d0ccce"
server_key_iv: "27f557bff19c829511912c4def934fc2"
server_key_version: "kuduclusterkey@0"
I20260619 08:20:34.772493 16028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:20:34.791149 16028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:20:34.793905 16028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:20:34.794167 16028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:20:34.794528 16028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:20:34.795032 16028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:20:34.795127 16028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:20:34.795257 16028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:20:34.795339 16028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:20:34.803210 16028 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:40521
I20260619 08:20:34.803758 16028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
I20260619 08:20:34.805519 16158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:40521 every 8 connection(s)
I20260619 08:20:34.808017 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 16028
I20260619 08:20:34.808132 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:20:34.808321 4702 external_mini_cluster.cc:1428] Setting key c7ae6fd98dd778af4b205fbe0afae6e4
Jun 19 08:20:34 dist-test-slave-d3n8 krb5kdc[15471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857234, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.151.130@KRBTEST.COM; defaulting to no policy
I20260619 08:20:34.825199 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:20:34.806370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:39353 (local address 127.4.151.190:43947)
0619 08:20:34.813579 (+ 7209us) server_negotiation.cc:206] Beginning negotiation
0619 08:20:34.813581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:20:34.813752 (+ 171us) server_negotiation.cc:407] Connection header received
0619 08:20:34.814537 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:20:34.814538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:20:34.814555 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:20:34.814648 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:20:34.815540 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:20:34.816425 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:20:34.818251 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:20:34.818595 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:20:34.818645 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:20:34.821482 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:20:34.821484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:20:34.821485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:20:34.821494 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:20:34.823427 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:20:34.823987 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:20:34.823988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:20:34.823989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:20:34.824070 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:20:34.824385 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:20:34.824386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:20:34.824386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:20:34.824535 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:20:34.824590 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 08:20:34.824975 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 08:20:34.825085 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7176,"thread_start_us":93,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.130@KRBTEST.COM" created.
I20260619 08:20:34.825918 16161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:20:34.806347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:43947 (local address 127.4.151.129:39353)
0619 08:20:34.813518 (+ 7171us) negotiation.cc:107] Waiting for socket to connect
0619 08:20:34.813602 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0619 08:20:34.814395 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:20:34.814807 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:20:34.814811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:20:34.815096 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:20:34.815406 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:20:34.815412 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:20:34.816589 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:20:34.816590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:20:34.818125 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:20:34.818129 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:20:34.818774 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:20:34.818775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:20:34.818933 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:20:34.819583 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:20:34.819595 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:20:34.821325 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:20:34.823534 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:20:34.823544 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:20:34.823547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:20:34.823894 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:20:34.824144 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:20:34.824145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:20:34.824145 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:20:34.824301 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:20:34.824624 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:20:34.824631 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:20:34.824806 (+ 175us) client_negotiation.cc:769] Sending connection context
0619 08:20:34.825475 (+ 669us) client_negotiation.cc:240] Negotiation successful
0619 08:20:34.825663 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7098,"thread_start_us":87,"threads_started":1}
I20260619 08:20:34.826670 16159 heartbeater.cc:347] Connected to a master server at 127.4.151.190:43947
I20260619 08:20:34.826787 16159 heartbeater.cc:464] Registering TS with master...
I20260619 08:20:34.827000 16159 heartbeater.cc:511] Master 127.4.151.190:43947 requested a full tablet report, sending...
I20260619 08:20:34.827513 15947 ts_manager.cc:194] Registered new tserver with Master: 352d9717c3d049128c48c41cdf7b5ba7 (127.4.151.129:40521)
I20260619 08:20:34.828536 15947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.129@KRBTEST.COM'} at 127.4.151.129:39353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
I20260619 08:20:34.857569 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.151.148:47465/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43947
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:20:34.965127 16165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:20:34.965373 16165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:20:34.965428 16165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:20:34.966642 16165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:20:34.966717 16165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:20:34.966818 16165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:20:34.968541 16165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.151.148:47465/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:20:34.968820 16165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:20:34.969086 16165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:20:34.974265 16175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:20:34.974421 16173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:20:34.974592 16172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:20:34.975543 16165 server_base.cc:1034] running on GCE node
I20260619 08:20:34.975801 16165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:20:34.976048 16165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:20:34.977198 16165 hybrid_clock.cc:648] HybridClock initialized: now 1781857234977185 us; error 38 us; skew 500 ppm
Jun 19 08:20:34 dist-test-slave-d3n8 krb5kdc[15471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857234, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:20:34.980414 16165 init.cc:377] Logged in from keytab as kudu/127.4.151.130@KRBTEST.COM (short username kudu)
I20260619 08:20:34.980945 16165 webserver.cc:460] Webserver started at http://127.4.151.130:35105/ using document root <none> and password file <none>
I20260619 08:20:34.981213 16165 fs_manager.cc:359] Metadata directory not provided
I20260619 08:20:34.981283 16165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:20:34.981449 16165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 08:20:34 dist-test-slave-d3n8 krb5kdc[15471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857234, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for HTTP/127.4.151.148@KRBTEST.COM
I20260619 08:20:34.997026 16165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "fc8367d53bf5486e8bee274848ca66c5"
format_stamp: "Formatted at 2026-06-19 08:20:34 on dist-test-slave-d3n8"
server_key: "b947d6ec509d3df5bcaef27e53a3bc24"
server_key_iv: "9d8499dc847e8e44098e8325aa8cf2e2"
server_key_version: "kuduclusterkey@0"
I20260619 08:20:34.997641 16165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "fc8367d53bf5486e8bee274848ca66c5"
format_stamp: "Formatted at 2026-06-19 08:20:34 on dist-test-slave-d3n8"
server_key: "b947d6ec509d3df5bcaef27e53a3bc24"
server_key_iv: "9d8499dc847e8e44098e8325aa8cf2e2"
server_key_version: "kuduclusterkey@0"
Jun 19 08:20:34 dist-test-slave-d3n8 krb5kdc[15471](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781857234, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:20:35.005322 16165 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.000s
I20260619 08:20:35.030910 16182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:20:35.031206 16165 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.000s
I20260619 08:20:35.031306 16165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "fc8367d53bf5486e8bee274848ca66c5"
format_stamp: "Formatted at 2026-06-19 08:20:34 on dist-test-slave-d3n8"
server_key: "b947d6ec509d3df5bcaef27e53a3bc24"
server_key_iv: "9d8499dc847e8e44098e8325aa8cf2e2"
server_key_version: "kuduclusterkey@0"
I20260619 08:20:35.031407 16165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:20:35.055354 16165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:20:35.058044 16165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:20:35.058288 16165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:20:35.058597 16165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:20:35.059046 16165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:20:35.059157 16165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:20:35.059235 16165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:20:35.059296 16165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:20:35.065212 16165 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:39237
I20260619 08:20:35.065714 16165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
I20260619 08:20:35.067624 16295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:39237 every 8 connection(s)
I20260619 08:20:35.069198 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 16165
I20260619 08:20:35.069305 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:20:35.069518 4702 external_mini_cluster.cc:1428] Setting key 936dfcc67ab717df9684d8547989960e
Jun 19 08:20:35 dist-test-slave-d3n8 krb5kdc[15471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857234, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:20:35.079681 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:20:35.068342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:43947 (local address 127.4.151.130:53171)
0619 08:20:35.068590 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 08:20:35.068629 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 08:20:35.069249 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:20:35.069624 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:20:35.069628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:20:35.069874 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:20:35.070152 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:20:35.070157 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.071165 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.071165 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:20:35.072424 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:20:35.072428 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.073025 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.073026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:20:35.073180 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:20:35.073893 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:20:35.073903 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:20:35.075835 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:20:35.077866 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:20:35.077875 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:20:35.077877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:20:35.078217 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:20:35.078453 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:20:35.078454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:20:35.078454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:20:35.078597 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:20:35.078899 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:20:35.078906 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:20:35.079111 (+ 205us) client_negotiation.cc:769] Sending connection context
0619 08:20:35.079240 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 08:20:35.079423 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":93,"threads_started":1}
I20260619 08:20:35.080351 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:20:35.068373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:53171 (local address 127.4.151.190:43947)
0619 08:20:35.068420 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 08:20:35.068423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:20:35.068727 (+ 304us) server_negotiation.cc:407] Connection header received
0619 08:20:35.069367 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:20:35.069392 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:20:35.069408 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:20:35.069440 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:20:35.070262 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.071049 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.072526 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.072852 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.072931 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:20:35.075965 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:20:35.075966 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:20:35.075966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:20:35.075974 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:20:35.077748 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:20:35.078322 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:20:35.078323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:20:35.078324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:20:35.078383 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:20:35.078665 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:20:35.078666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:20:35.078667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:20:35.078814 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:20:35.078874 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 08:20:35.080149 (+ 1275us) server_negotiation.cc:299] Negotiation successful
0619 08:20:35.080257 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for kudu/127.4.151.131@KRBTEST.COM; defaulting to no policy
I20260619 08:20:35.081471 16296 heartbeater.cc:347] Connected to a master server at 127.4.151.190:43947
I20260619 08:20:35.081632 16296 heartbeater.cc:464] Registering TS with master...
I20260619 08:20:35.081874 16296 heartbeater.cc:511] Master 127.4.151.190:43947 requested a full tablet report, sending...
I20260619 08:20:35.082286 15947 ts_manager.cc:194] Registered new tserver with Master: fc8367d53bf5486e8bee274848ca66c5 (127.4.151.130:39237)
I20260619 08:20:35.082803 15947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.130@KRBTEST.COM'} at 127.4.151.130:53171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
I20260619 08:20:35.106753 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.151.148:47465/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43947
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:20:35.212500 16302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:20:35.212878 16302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:20:35.212965 16302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:20:35.214815 16302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:20:35.214931 16302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:20:35.215020 16302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:20:35.217257 16302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.151.148:47465/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:43947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:20:35.217753 16302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:20:35.218082 16302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:20:35.223263 16308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:20:35.223361 16307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:20:35.223619 16310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:20:35.224143 16302 server_base.cc:1034] running on GCE node
I20260619 08:20:35.224371 16302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:20:35.224624 16302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:20:35.225764 16302 hybrid_clock.cc:648] HybridClock initialized: now 1781857235225735 us; error 42 us; skew 500 ppm
Jun 19 08:20:35 dist-test-slave-d3n8 krb5kdc[15471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857235, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:20:35.228644 16302 init.cc:377] Logged in from keytab as kudu/127.4.151.131@KRBTEST.COM (short username kudu)
I20260619 08:20:35.229151 16302 webserver.cc:460] Webserver started at http://127.4.151.131:46697/ using document root <none> and password file <none>
I20260619 08:20:35.229466 16302 fs_manager.cc:359] Metadata directory not provided
I20260619 08:20:35.229538 16302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:20:35.229687 16302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 08:20:35 dist-test-slave-d3n8 krb5kdc[15471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857235, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for HTTP/127.4.151.148@KRBTEST.COM
I20260619 08:20:35.243413 16302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "58785c60ad0e4351bfab46b2a3e2c8e5"
format_stamp: "Formatted at 2026-06-19 08:20:35 on dist-test-slave-d3n8"
server_key: "af55c11c8f7942f1fa2b63df691824d6"
server_key_iv: "0d0f13fef83d796b8778889f975628fd"
server_key_version: "kuduclusterkey@0"
I20260619 08:20:35.243884 16302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "58785c60ad0e4351bfab46b2a3e2c8e5"
format_stamp: "Formatted at 2026-06-19 08:20:35 on dist-test-slave-d3n8"
server_key: "af55c11c8f7942f1fa2b63df691824d6"
server_key_iv: "0d0f13fef83d796b8778889f975628fd"
server_key_version: "kuduclusterkey@0"
I20260619 08:20:35.253891 16302 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.002s sys 0.002s
I20260619 08:20:35.281525 16317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:20:35.281782 16302 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260619 08:20:35.281903 16302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "58785c60ad0e4351bfab46b2a3e2c8e5"
format_stamp: "Formatted at 2026-06-19 08:20:35 on dist-test-slave-d3n8"
server_key: "af55c11c8f7942f1fa2b63df691824d6"
server_key_iv: "0d0f13fef83d796b8778889f975628fd"
server_key_version: "kuduclusterkey@0"
I20260619 08:20:35.282052 16302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:20:35.304242 16302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:20:35.307330 16302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:20:35.307577 16302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:20:35.308025 16302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:20:35.308506 16302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:20:35.308635 16302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:20:35.308736 16302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:20:35.308806 16302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:20:35.316546 16302 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:42715
I20260619 08:20:35.316905 16430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:42715 every 8 connection(s)
I20260619 08:20:35.317090 16302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
Jun 19 08:20:35 dist-test-slave-d3n8 krb5kdc[15471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857235, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:20:35.326784 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 16302
I20260619 08:20:35.326893 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:20:35.327071 4702 external_mini_cluster.cc:1428] Setting key 857feb36a55368dbd00149f543320efc
I20260619 08:20:35.328327 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:20:35.318383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:37899 (local address 127.4.151.190:43947)
0619 08:20:35.318434 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 08:20:35.318436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:20:35.318654 (+ 218us) server_negotiation.cc:407] Connection header received
0619 08:20:35.319358 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:20:35.319360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:20:35.319375 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:20:35.319474 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:20:35.320200 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.321045 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.322233 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.322459 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.322545 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:20:35.325199 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:20:35.325200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:20:35.325201 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:20:35.325210 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:20:35.326763 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:20:35.327282 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:20:35.327283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:20:35.327283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:20:35.327343 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:20:35.327596 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:20:35.327597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:20:35.327598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:20:35.327751 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:20:35.327815 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 08:20:35.328150 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 08:20:35.328233 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 08:20:35.328536 16433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:20:35.318260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:43947 (local address 127.4.151.131:37899)
0619 08:20:35.318501 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 08:20:35.318542 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 08:20:35.319235 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:20:35.319545 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:20:35.319549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:20:35.319808 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:20:35.320079 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:20:35.320085 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.321167 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.321168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:20:35.322102 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:20:35.322105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.322567 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.322568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:20:35.322707 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:20:35.323405 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:20:35.323415 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:20:35.325076 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:20:35.326861 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:20:35.326869 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:20:35.326871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:20:35.327182 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:20:35.327413 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:20:35.327414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:20:35.327414 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:20:35.327540 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:20:35.327819 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:20:35.327826 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:20:35.328012 (+ 186us) client_negotiation.cc:769] Sending connection context
0619 08:20:35.328129 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 08:20:35.328285 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":83,"threads_started":1}
I20260619 08:20:35.328989 16431 heartbeater.cc:347] Connected to a master server at 127.4.151.190:43947
I20260619 08:20:35.329069 16431 heartbeater.cc:464] Registering TS with master...
I20260619 08:20:35.329277 16431 heartbeater.cc:511] Master 127.4.151.190:43947 requested a full tablet report, sending...
I20260619 08:20:35.329695 15947 ts_manager.cc:194] Registered new tserver with Master: 58785c60ad0e4351bfab46b2a3e2c8e5 (127.4.151.131:42715)
I20260619 08:20:35.330183 15947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.131@KRBTEST.COM'} at 127.4.151.131:37899
I20260619 08:20:35.339329 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:20:35.347910 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:20:35.340856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46074 (local address 127.4.151.190:43947)
0619 08:20:35.340904 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 08:20:35.340906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:20:35.341091 (+ 185us) server_negotiation.cc:407] Connection header received
0619 08:20:35.341119 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:20:35.341119 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:20:35.341129 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:20:35.341199 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:20:35.341931 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.342614 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.343372 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.343594 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.343687 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:20:35.345146 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:20:35.345147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:20:35.345148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:20:35.345155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:20:35.346662 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:20:35.347133 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:20:35.347134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:20:35.347134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:20:35.347193 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:20:35.347392 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:20:35.347393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:20:35.347393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:20:35.347520 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:20:35.347624 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 08:20:35.347723 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 08:20:35.347801 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:20:35.354357 15947 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:20:35.356220 15947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:20:35.365309 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:20:35.359051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:40521 (local address 127.0.0.1:48314)
0619 08:20:35.360237 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0619 08:20:35.360247 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:20:35.360291 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:20:35.360599 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:20:35.360602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:20:35.360603 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:20:35.360670 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:20:35.360672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.362286 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.362286 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:20:35.363421 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:20:35.363424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.364903 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.364904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:20:35.365001 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:20:35.365015 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 08:20:35.365086 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 08:20:35.365140 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1171,"mutex_wait_us":4,"thread_start_us":84,"threads_started":1}
I20260619 08:20:35.370060 16447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:20:35.359547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48314 (local address 127.4.151.129:40521)
0619 08:20:35.360388 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0619 08:20:35.360392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:20:35.360401 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:20:35.360434 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:20:35.360436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:20:35.360452 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:20:35.360520 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:20:35.360750 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.362202 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.363523 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.364734 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.364807 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:20:35.364868 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 08:20:35.369910 (+ 5042us) server_negotiation.cc:299] Negotiation successful
0619 08:20:35.369977 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":811,"thread_start_us":52,"threads_started":1}
I20260619 08:20:35.371425 16444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:20:35.359049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:39237 (local address 127.0.0.1:55216)
0619 08:20:35.359530 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 08:20:35.359572 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 08:20:35.359642 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:20:35.360730 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:20:35.360731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:20:35.360733 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:20:35.360786 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:20:35.360787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.361998 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.361999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:20:35.369908 (+ 7909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:20:35.369911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.371149 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.371151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:20:35.371225 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:20:35.371229 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:20:35.371290 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 08:20:35.371332 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":88,"threads_started":1}
I20260619 08:20:35.371392 16093 tablet_service.cc:1467] Processing CreateTablet for tablet d4a81bdec80d476c8cd461b8991bd801 (DEFAULT_TABLE table=test-table [id=dfa1981235cc476c9d5b4c59d75c0a1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:20:35.371814 16093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4a81bdec80d476c8cd461b8991bd801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:20:35.371862 16446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:20:35.359325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55216 (local address 127.4.151.130:39237)
0619 08:20:35.359807 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0619 08:20:35.359813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 08:20:35.359823 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:20:35.359862 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:20:35.359864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:20:35.359881 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:20:35.359952 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:20:35.360862 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.361915 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.370176 (+ 8261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.370973 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.371047 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:20:35.371116 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 08:20:35.371725 (+ 609us) server_negotiation.cc:299] Negotiation successful
0619 08:20:35.371777 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":70,"threads_started":1}
I20260619 08:20:35.372401 16211 tablet_service.cc:1467] Processing CreateTablet for tablet d4a81bdec80d476c8cd461b8991bd801 (DEFAULT_TABLE table=test-table [id=dfa1981235cc476c9d5b4c59d75c0a1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:20:35.372712 16211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4a81bdec80d476c8cd461b8991bd801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:20:35.376461 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:20:35.359271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:42715 (local address 127.0.0.1:33942)
0619 08:20:35.365787 (+ 6516us) negotiation.cc:107] Waiting for socket to connect
0619 08:20:35.365797 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:20:35.365833 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:20:35.366114 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:20:35.366115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:20:35.366117 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:20:35.366149 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:20:35.366150 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.367772 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.367772 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:20:35.368777 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:20:35.368779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.376209 (+ 7430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.376210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:20:35.376280 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:20:35.376285 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:20:35.376341 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 08:20:35.376387 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6503,"thread_start_us":69,"threads_started":1}
I20260619 08:20:35.376852 16448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:20:35.359703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33942 (local address 127.4.151.131:42715)
0619 08:20:35.360187 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0619 08:20:35.360191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:20:35.365896 (+ 5705us) server_negotiation.cc:407] Connection header received
0619 08:20:35.365947 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:20:35.365949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:20:35.365966 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:20:35.366034 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:20:35.366236 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.367653 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.368897 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.369680 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.369748 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:20:35.369815 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 08:20:35.376705 (+ 6890us) server_negotiation.cc:299] Negotiation successful
0619 08:20:35.376764 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":75,"threads_started":1}
I20260619 08:20:35.377766 16365 tablet_service.cc:1467] Processing CreateTablet for tablet d4a81bdec80d476c8cd461b8991bd801 (DEFAULT_TABLE table=test-table [id=dfa1981235cc476c9d5b4c59d75c0a1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:20:35.377970 16450 tablet_bootstrap.cc:492] T d4a81bdec80d476c8cd461b8991bd801 P fc8367d53bf5486e8bee274848ca66c5: Bootstrap starting.
I20260619 08:20:35.378089 16365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4a81bdec80d476c8cd461b8991bd801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:20:35.379149 16450 tablet_bootstrap.cc:654] T d4a81bdec80d476c8cd461b8991bd801 P fc8367d53bf5486e8bee274848ca66c5: Neither blocks nor log segments found. Creating new log.
I20260619 08:20:35.379451 16450 log.cc:824] T d4a81bdec80d476c8cd461b8991bd801 P fc8367d53bf5486e8bee274848ca66c5: Log is configured to *not* fsync() on all Append() calls
I20260619 08:20:35.382376 16449 tablet_bootstrap.cc:492] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7: Bootstrap starting.
I20260619 08:20:35.383181 16449 tablet_bootstrap.cc:654] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7: Neither blocks nor log segments found. Creating new log.
I20260619 08:20:35.383562 16449 log.cc:824] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7: Log is configured to *not* fsync() on all Append() calls
I20260619 08:20:35.383773 16452 tablet_bootstrap.cc:492] T d4a81bdec80d476c8cd461b8991bd801 P 58785c60ad0e4351bfab46b2a3e2c8e5: Bootstrap starting.
I20260619 08:20:35.384470 16450 tablet_bootstrap.cc:492] T d4a81bdec80d476c8cd461b8991bd801 P fc8367d53bf5486e8bee274848ca66c5: No bootstrap required, opened a new log
I20260619 08:20:35.384557 16450 ts_tablet_manager.cc:1397] T d4a81bdec80d476c8cd461b8991bd801 P fc8367d53bf5486e8bee274848ca66c5: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260619 08:20:35.384868 16452 tablet_bootstrap.cc:654] T d4a81bdec80d476c8cd461b8991bd801 P 58785c60ad0e4351bfab46b2a3e2c8e5: Neither blocks nor log segments found. Creating new log.
I20260619 08:20:35.385138 16452 log.cc:824] T d4a81bdec80d476c8cd461b8991bd801 P 58785c60ad0e4351bfab46b2a3e2c8e5: Log is configured to *not* fsync() on all Append() calls
I20260619 08:20:35.386269 16452 tablet_bootstrap.cc:492] T d4a81bdec80d476c8cd461b8991bd801 P 58785c60ad0e4351bfab46b2a3e2c8e5: No bootstrap required, opened a new log
I20260619 08:20:35.386329 16452 ts_tablet_manager.cc:1397] T d4a81bdec80d476c8cd461b8991bd801 P 58785c60ad0e4351bfab46b2a3e2c8e5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 08:20:35.386843 16450 raft_consensus.cc:348] T d4a81bdec80d476c8cd461b8991bd801 P fc8367d53bf5486e8bee274848ca66c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352d9717c3d049128c48c41cdf7b5ba7" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 40521 } } peers { permanent_uuid: "fc8367d53bf5486e8bee274848ca66c5" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39237 } } peers { permanent_uuid: "58785c60ad0e4351bfab46b2a3e2c8e5" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42715 } }
I20260619 08:20:35.387013 16450 raft_consensus.cc:374] T d4a81bdec80d476c8cd461b8991bd801 P fc8367d53bf5486e8bee274848ca66c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:20:35.387045 16450 raft_consensus.cc:729] T d4a81bdec80d476c8cd461b8991bd801 P fc8367d53bf5486e8bee274848ca66c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8367d53bf5486e8bee274848ca66c5, State: Initialized, Role: FOLLOWER
I20260619 08:20:35.387364 16450 consensus_queue.cc:260] T d4a81bdec80d476c8cd461b8991bd801 P fc8367d53bf5486e8bee274848ca66c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352d9717c3d049128c48c41cdf7b5ba7" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 40521 } } peers { permanent_uuid: "fc8367d53bf5486e8bee274848ca66c5" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39237 } } peers { permanent_uuid: "58785c60ad0e4351bfab46b2a3e2c8e5" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42715 } }
I20260619 08:20:35.387732 16450 ts_tablet_manager.cc:1428] T d4a81bdec80d476c8cd461b8991bd801 P fc8367d53bf5486e8bee274848ca66c5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:20:35.387919 16296 heartbeater.cc:503] Master 127.4.151.190:43947 was elected leader, sending a full tablet report...
I20260619 08:20:35.388846 16452 raft_consensus.cc:348] T d4a81bdec80d476c8cd461b8991bd801 P 58785c60ad0e4351bfab46b2a3e2c8e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352d9717c3d049128c48c41cdf7b5ba7" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 40521 } } peers { permanent_uuid: "fc8367d53bf5486e8bee274848ca66c5" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39237 } } peers { permanent_uuid: "58785c60ad0e4351bfab46b2a3e2c8e5" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42715 } }
I20260619 08:20:35.389086 16452 raft_consensus.cc:374] T d4a81bdec80d476c8cd461b8991bd801 P 58785c60ad0e4351bfab46b2a3e2c8e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:20:35.389195 16452 raft_consensus.cc:729] T d4a81bdec80d476c8cd461b8991bd801 P 58785c60ad0e4351bfab46b2a3e2c8e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58785c60ad0e4351bfab46b2a3e2c8e5, State: Initialized, Role: FOLLOWER
I20260619 08:20:35.389407 16452 consensus_queue.cc:260] T d4a81bdec80d476c8cd461b8991bd801 P 58785c60ad0e4351bfab46b2a3e2c8e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352d9717c3d049128c48c41cdf7b5ba7" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 40521 } } peers { permanent_uuid: "fc8367d53bf5486e8bee274848ca66c5" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39237 } } peers { permanent_uuid: "58785c60ad0e4351bfab46b2a3e2c8e5" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42715 } }
I20260619 08:20:35.389752 16452 ts_tablet_manager.cc:1428] T d4a81bdec80d476c8cd461b8991bd801 P 58785c60ad0e4351bfab46b2a3e2c8e5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:20:35.389925 16431 heartbeater.cc:503] Master 127.4.151.190:43947 was elected leader, sending a full tablet report...
I20260619 08:20:35.390295 16449 tablet_bootstrap.cc:492] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7: No bootstrap required, opened a new log
I20260619 08:20:35.390368 16449 ts_tablet_manager.cc:1397] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260619 08:20:35.392508 16449 raft_consensus.cc:348] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352d9717c3d049128c48c41cdf7b5ba7" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 40521 } } peers { permanent_uuid: "fc8367d53bf5486e8bee274848ca66c5" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39237 } } peers { permanent_uuid: "58785c60ad0e4351bfab46b2a3e2c8e5" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42715 } }
I20260619 08:20:35.392642 16449 raft_consensus.cc:374] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:20:35.392670 16449 raft_consensus.cc:729] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 352d9717c3d049128c48c41cdf7b5ba7, State: Initialized, Role: FOLLOWER
I20260619 08:20:35.392786 16449 consensus_queue.cc:260] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352d9717c3d049128c48c41cdf7b5ba7" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 40521 } } peers { permanent_uuid: "fc8367d53bf5486e8bee274848ca66c5" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39237 } } peers { permanent_uuid: "58785c60ad0e4351bfab46b2a3e2c8e5" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42715 } }
I20260619 08:20:35.393054 16449 ts_tablet_manager.cc:1428] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:20:35.393093 16159 heartbeater.cc:503] Master 127.4.151.190:43947 was elected leader, sending a full tablet report...
Jun 19 08:20:35 dist-test-slave-d3n8 krb5kdc[15471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857234, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for HTTP/127.4.151.148@KRBTEST.COM
I20260619 08:20:35.512969 16457 raft_consensus.cc:482] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:20:35.513146 16457 raft_consensus.cc:504] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352d9717c3d049128c48c41cdf7b5ba7" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 40521 } } peers { permanent_uuid: "fc8367d53bf5486e8bee274848ca66c5" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39237 } } peers { permanent_uuid: "58785c60ad0e4351bfab46b2a3e2c8e5" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42715 } }
I20260619 08:20:35.513604 16457 leader_election.cc:288] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc8367d53bf5486e8bee274848ca66c5 (127.4.151.130:39237), 58785c60ad0e4351bfab46b2a3e2c8e5 (127.4.151.131:42715)
I20260619 08:20:35.520913 16448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:20:35.514288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:54955 (local address 127.4.151.131:42715)
0619 08:20:35.514333 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 08:20:35.514336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:20:35.514981 (+ 645us) server_negotiation.cc:407] Connection header received
0619 08:20:35.515250 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:20:35.515251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:20:35.515262 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:20:35.515295 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:20:35.515565 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.516259 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.517793 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.518332 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.518387 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:20:35.518398 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:20:35.520778 (+ 2380us) server_negotiation.cc:299] Negotiation successful
0619 08:20:35.520828 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:20:35.523552 16459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:20:35.513903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:42715 (local address 127.4.151.129:54955)
0619 08:20:35.514917 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0619 08:20:35.514929 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:20:35.515012 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:20:35.515380 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:20:35.515381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:20:35.515383 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:20:35.515477 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:20:35.515480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.516373 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.516374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:20:35.517686 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:20:35.517689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.520610 (+ 2921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.520611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:20:35.520689 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:20:35.520693 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:20:35.523389 (+ 2696us) client_negotiation.cc:240] Negotiation successful
0619 08:20:35.523453 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"thread_start_us":128,"threads_started":1}
I20260619 08:20:35.524125 16385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4a81bdec80d476c8cd461b8991bd801" candidate_uuid: "352d9717c3d049128c48c41cdf7b5ba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58785c60ad0e4351bfab46b2a3e2c8e5" is_pre_election: true
I20260619 08:20:35.524319 16385 raft_consensus.cc:2454] T d4a81bdec80d476c8cd461b8991bd801 P 58785c60ad0e4351bfab46b2a3e2c8e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 352d9717c3d049128c48c41cdf7b5ba7 in term 0.
I20260619 08:20:35.524605 16048 leader_election.cc:302] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 352d9717c3d049128c48c41cdf7b5ba7, 58785c60ad0e4351bfab46b2a3e2c8e5; no voters:
I20260619 08:20:35.524777 16457 raft_consensus.cc:2781] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:20:35.524828 16457 raft_consensus.cc:482] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:20:35.524852 16457 raft_consensus.cc:3037] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:20:35.525738 16446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:20:35.514656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:51755 (local address 127.4.151.130:39237)
0619 08:20:35.514696 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 08:20:35.514698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:20:35.521562 (+ 6864us) server_negotiation.cc:407] Connection header received
0619 08:20:35.521687 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:20:35.521688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:20:35.521702 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:20:35.521734 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:20:35.521949 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.522716 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.524703 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.525292 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.525345 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:20:35.525353 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 08:20:35.525601 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 08:20:35.525652 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:20:35.525879 16457 raft_consensus.cc:504] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352d9717c3d049128c48c41cdf7b5ba7" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 40521 } } peers { permanent_uuid: "fc8367d53bf5486e8bee274848ca66c5" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39237 } } peers { permanent_uuid: "58785c60ad0e4351bfab46b2a3e2c8e5" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42715 } }
I20260619 08:20:35.526031 16457 leader_election.cc:288] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7 [CANDIDATE]: Term 1 election: Requested vote from peers fc8367d53bf5486e8bee274848ca66c5 (127.4.151.130:39237), 58785c60ad0e4351bfab46b2a3e2c8e5 (127.4.151.131:42715)
I20260619 08:20:35.526620 16458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:20:35.514456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:39237 (local address 127.4.151.129:51755)
0619 08:20:35.521466 (+ 7010us) negotiation.cc:107] Waiting for socket to connect
0619 08:20:35.521480 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 08:20:35.521596 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:20:35.521798 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:20:35.521798 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:20:35.521800 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:20:35.521877 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:20:35.521879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.522859 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.522860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:20:35.524582 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:20:35.524586 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.525446 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.525447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:20:35.525521 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:20:35.525525 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:20:35.525823 (+ 298us) client_negotiation.cc:240] Negotiation successful
0619 08:20:35.525869 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6986,"thread_start_us":78,"threads_started":1}
I20260619 08:20:35.526582 16385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4a81bdec80d476c8cd461b8991bd801" candidate_uuid: "352d9717c3d049128c48c41cdf7b5ba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58785c60ad0e4351bfab46b2a3e2c8e5"
I20260619 08:20:35.526678 16385 raft_consensus.cc:3037] T d4a81bdec80d476c8cd461b8991bd801 P 58785c60ad0e4351bfab46b2a3e2c8e5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:20:35.527139 16250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4a81bdec80d476c8cd461b8991bd801" candidate_uuid: "352d9717c3d049128c48c41cdf7b5ba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8367d53bf5486e8bee274848ca66c5" is_pre_election: true
I20260619 08:20:35.527290 16250 raft_consensus.cc:2454] T d4a81bdec80d476c8cd461b8991bd801 P fc8367d53bf5486e8bee274848ca66c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 352d9717c3d049128c48c41cdf7b5ba7 in term 0.
I20260619 08:20:35.527422 16249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4a81bdec80d476c8cd461b8991bd801" candidate_uuid: "352d9717c3d049128c48c41cdf7b5ba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8367d53bf5486e8bee274848ca66c5"
I20260619 08:20:35.527483 16249 raft_consensus.cc:3037] T d4a81bdec80d476c8cd461b8991bd801 P fc8367d53bf5486e8bee274848ca66c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:20:35.527575 16385 raft_consensus.cc:2454] T d4a81bdec80d476c8cd461b8991bd801 P 58785c60ad0e4351bfab46b2a3e2c8e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 352d9717c3d049128c48c41cdf7b5ba7 in term 1.
I20260619 08:20:35.527949 16048 leader_election.cc:302] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 352d9717c3d049128c48c41cdf7b5ba7, 58785c60ad0e4351bfab46b2a3e2c8e5; no voters:
I20260619 08:20:35.528069 16457 raft_consensus.cc:2781] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:20:35.528210 16457 raft_consensus.cc:686] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7 [term 1 LEADER]: Becoming Leader. State: Replica: 352d9717c3d049128c48c41cdf7b5ba7, State: Running, Role: LEADER
I20260619 08:20:35.528301 16457 consensus_queue.cc:237] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352d9717c3d049128c48c41cdf7b5ba7" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 40521 } } peers { permanent_uuid: "fc8367d53bf5486e8bee274848ca66c5" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39237 } } peers { permanent_uuid: "58785c60ad0e4351bfab46b2a3e2c8e5" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42715 } }
I20260619 08:20:35.528347 16249 raft_consensus.cc:2454] T d4a81bdec80d476c8cd461b8991bd801 P fc8367d53bf5486e8bee274848ca66c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 352d9717c3d049128c48c41cdf7b5ba7 in term 1.
I20260619 08:20:35.529156 15947 catalog_manager.cc:5495] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 352d9717c3d049128c48c41cdf7b5ba7 (127.4.151.129). New cstate: current_term: 1 leader_uuid: "352d9717c3d049128c48c41cdf7b5ba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352d9717c3d049128c48c41cdf7b5ba7" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 40521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc8367d53bf5486e8bee274848ca66c5" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58785c60ad0e4351bfab46b2a3e2c8e5" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42715 } health_report { overall_health: UNKNOWN } } }
I20260619 08:20:35.553807 16447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:20:35.550517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48326 (local address 127.4.151.129:40521)
0619 08:20:35.550565 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 08:20:35.550567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:20:35.550578 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:20:35.550622 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:20:35.550623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:20:35.550641 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:20:35.550708 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:20:35.550878 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.551609 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.552659 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:20:35.552931 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:20:35.552984 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:20:35.553205 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:20:35.553336 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:20:35.553442 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 08:20:35.553685 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 08:20:35.553706 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 08:20:35.556226 16160 tablet.cc:2369] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:20:35.559949 16249 raft_consensus.cc:1261] T d4a81bdec80d476c8cd461b8991bd801 P fc8367d53bf5486e8bee274848ca66c5 [term 1 FOLLOWER]: Refusing update from remote peer 352d9717c3d049128c48c41cdf7b5ba7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:20:35.560245 16385 raft_consensus.cc:1261] T d4a81bdec80d476c8cd461b8991bd801 P 58785c60ad0e4351bfab46b2a3e2c8e5 [term 1 FOLLOWER]: Refusing update from remote peer 352d9717c3d049128c48c41cdf7b5ba7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:20:35.560309 16457 consensus_queue.cc:1035] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc8367d53bf5486e8bee274848ca66c5" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:20:35.560515 16457 consensus_queue.cc:1035] T d4a81bdec80d476c8cd461b8991bd801 P 352d9717c3d049128c48c41cdf7b5ba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58785c60ad0e4351bfab46b2a3e2c8e5" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:20:35.567229 16463 mvcc.cc:205] Tried to move back new op lower bound from 7298487236850311168 to 7298487236724486144. Current Snapshot: MvccSnapshot[applied={T|T < 7298487236850311168}]
I20260619 08:20:35.567374 16466 mvcc.cc:205] Tried to move back new op lower bound from 7298487236850311168 to 7298487236724486144. Current Snapshot: MvccSnapshot[applied={T|T < 7298487236850311168}]
I20260619 08:20:35.569072 16464 mvcc.cc:205] Tried to move back new op lower bound from 7298487236850311168 to 7298487236724486144. Current Snapshot: MvccSnapshot[applied={T|T < 7298487236850311168 or (T in {7298487236850311168})}]
I20260619 08:20:35.570931 15946 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:20:35.571035 15946 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:20:35.571762 15946 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 86d37d9c85734c64bd6f156975586234: Sending DeleteTablet for 3 replicas of tablet d4a81bdec80d476c8cd461b8991bd801
I20260619 08:20:35.572091 16211 tablet_service.cc:1514] Processing DeleteTablet for tablet d4a81bdec80d476c8cd461b8991bd801 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:20:35 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:55216
I20260619 08:20:35.572513 16093 tablet_service.cc:1514] Processing DeleteTablet for tablet d4a81bdec80d476c8cd461b8991bd801 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:20:35 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:48314
I20260619 08:20:35.572786 16365 tablet_service.cc:1514] Processing DeleteTablet for tablet d4a81bdec80d476c8cd461b8991bd801 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:20:35 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:33942
I20260619 08:20:35.573297 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 16028
I20260619 08:20:35.575030 16474 tablet_replica.cc:330] T d4a81bdec80d476c8cd461b8991bd801 P 58785c60ad0e4351bfab46b2a3e2c8e5: stopping tablet replica
I20260619 08:20:35.575160 16474 raft_consensus.cc:2229] T d4a81bdec80d476c8cd461b8991bd801 P 58785c60ad0e4351bfab46b2a3e2c8e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:20:35.575279 16474 raft_consensus.cc:2258] T d4a81bdec80d476c8cd461b8991bd801 P 58785c60ad0e4351bfab46b2a3e2c8e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:20:35.575448 16472 tablet_replica.cc:330] T d4a81bdec80d476c8cd461b8991bd801 P fc8367d53bf5486e8bee274848ca66c5: stopping tablet replica
I20260619 08:20:35.575546 16472 raft_consensus.cc:2229] T d4a81bdec80d476c8cd461b8991bd801 P fc8367d53bf5486e8bee274848ca66c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:20:35.575644 16472 raft_consensus.cc:2258] T d4a81bdec80d476c8cd461b8991bd801 P fc8367d53bf5486e8bee274848ca66c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:20:35.576074 16472 ts_tablet_manager.cc:1905] T d4a81bdec80d476c8cd461b8991bd801 P fc8367d53bf5486e8bee274848ca66c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:20:35.577912 16474 ts_tablet_manager.cc:1905] T d4a81bdec80d476c8cd461b8991bd801 P 58785c60ad0e4351bfab46b2a3e2c8e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:20:35.579334 16474 ts_tablet_manager.cc:1918] T d4a81bdec80d476c8cd461b8991bd801 P 58785c60ad0e4351bfab46b2a3e2c8e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:20:35.579409 16474 log.cc:1196] T d4a81bdec80d476c8cd461b8991bd801 P 58785c60ad0e4351bfab46b2a3e2c8e5: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/d4a81bdec80d476c8cd461b8991bd801
I20260619 08:20:35.579969 16474 ts_tablet_manager.cc:1939] T d4a81bdec80d476c8cd461b8991bd801 P 58785c60ad0e4351bfab46b2a3e2c8e5: Deleting consensus metadata
I20260619 08:20:35.580370 15933 catalog_manager.cc:4841] TS 58785c60ad0e4351bfab46b2a3e2c8e5 (127.4.151.131:42715): tablet d4a81bdec80d476c8cd461b8991bd801 (table test-table [id=dfa1981235cc476c9d5b4c59d75c0a1b]) successfully deleted
W20260619 08:20:35.580729 15933 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.151.129:40521 (error 108)
W20260619 08:20:35.581283 16445 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:20:35.581140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:40521 (local address 127.0.0.1:48342)
0619 08:20:35.581203 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 08:20:35.581245 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.129:40521: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 08:20:35.581411 15933 catalog_manager.cc:4584] TS 352d9717c3d049128c48c41cdf7b5ba7 (127.4.151.129:40521): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4a81bdec80d476c8cd461b8991bd801: Network error: Client connection negotiation failed: client connection to 127.4.151.129:40521: connect: Connection reset by peer (error 104)
I20260619 08:20:35.581748 16472 ts_tablet_manager.cc:1918] T d4a81bdec80d476c8cd461b8991bd801 P fc8367d53bf5486e8bee274848ca66c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:20:35.581750 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 16165
I20260619 08:20:35.585264 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:20:35.585149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:39237 (local address 127.0.0.1:55230)
0619 08:20:35.585205 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 08:20:35.585232 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.130:39237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 08:20:35.585613 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 16302
I20260619 08:20:35.590363 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 15915
I20260619 08:20:35.595732 4702 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 08:20:35.940702 4702 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 08:20:35.940827 4702 mini_ranger.cc:66] Stopping Ranger...
I20260619 08:20:36.285578 4702 mini_ranger.cc:68] Stopped Ranger
2026-06-19 08:20:36.285 UTC [15501] LOG: received smart shutdown request
2026-06-19 08:20:36.289 UTC [15501] LOG: background worker "logical replication launcher" (PID 15510) exited with exit code 1
2026-06-19 08:20:36.290 UTC [15505] LOG: shutting down
2026-06-19 08:20:36.488 UTC [15501] LOG: database system is shut down
2026-06-19T08:20:36Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48473 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:20:36 dist-test-slave-d3n8 krb5kdc[16484](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:20:36 dist-test-slave-d3n8 krb5kdc[16484](info): set up 2 sockets
Jun 19 08:20:36 dist-test-slave-d3n8 krb5kdc[16484](info): commencing operation
krb5kdc: starting...
W20260619 08:20:37.653515 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:20:37 dist-test-slave-d3n8 krb5kdc[16484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:20:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:20:37Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.151.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.151.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.151.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/rangeradmin_127.4.151.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.151.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/rangeradmin_127.4.151.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.151.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.151.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.151.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/rangerlookup_127.4.151.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.151.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/rangerlookup_127.4.151.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.151.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.151.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.151.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/HTTP_127.4.151.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.151.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/HTTP_127.4.151.148@KRBTEST.COM.keytab.
I20260619 08:20:37.829133 4702 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskR4iS53/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/postgres -l logfile start
2026-06-19 08:20:40.657 UTC [16514] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 08:20:40.657 UTC [16514] LOG: listening on IPv4 address "127.4.151.148", port 44785
2026-06-19 08:20:40.662 UTC [16514] LOG: listening on Unix socket "/tmp/.s.PGSQL.44785"
2026-06-19 08:20:40.685 UTC [16517] LOG: database system was shut down at 2026-06-19 08:20:38 UTC
2026-06-19 08:20:40.689 UTC [16514] LOG: database system is ready to accept connections
I20260619 08:20:41.649248 4702 mini_postgres.cc:96] Postgres bound to 44785
2026-06-19 08:20:41.655 UTC [16525] FATAL: database "slave" does not exist
127.4.151.148:44785 - accepting connections
I20260619 08:20:41.656126 4702 mini_ranger.cc:161] Starting Ranger...
I20260619 08:20:41.671939 4702 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 08:20:42.268667 4702 mini_ranger.cc:87] Created ranger Postgres database
I20260619 08:20:42.268760 4702 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ranger-admin
2026-06-19 08:20:42,541 [I] DB FLAVOR :POSTGRES
2026-06-19 08:20:42,541 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 08:20:42,541 [I] Checking connection..
2026-06-19 08:20:42,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:44785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 08:20:42,795 [I] Checking connection passed.
2026-06-19 08:20:42,795 [I] --------- Verifying version history table ---------
2026-06-19 08:20:42,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:44785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 08:20:43,052 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 08:20:43,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:44785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 08:20:43,318 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 08:20:43,319 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 08:20:43,319 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:44785/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 08:20:43.576 UTC [16589] WARNING: there is no transaction in progress
2026-06-19 08:20:43,583 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 08:20:43,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:44785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 08:20:43,844 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 08:20:43,844 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 08:20:43,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:44785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 08:20:44,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:44785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 08:20:44,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:44785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-d3n8.c.gcp-upstream.internal','N') ;"
2026-06-19 08:20:44,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:44785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 08:20:44,880 [I] Table x_portal_user does not exist in database ranger
2026-06-19 08:20:44,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:44785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 08:20:45,138 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 08:20:45,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:44785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 08:20:45,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:44785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 08:20:45,657 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 08:20:45,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:44785/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 08:20:46.719 UTC [16697] WARNING: there is no transaction in progress
2026-06-19 08:20:46.738 UTC [16697] WARNING: there is no transaction in progress
2026-06-19 08:20:46.758 UTC [16697] WARNING: there is no transaction in progress
2026-06-19 08:20:46.777 UTC [16697] WARNING: there is no transaction in progress
2026-06-19 08:20:46.799 UTC [16697] WARNING: there is no transaction in progress
2026-06-19 08:20:46.821 UTC [16697] WARNING: there is no transaction in progress
2026-06-19 08:20:46.886 UTC [16697] WARNING: there is no transaction in progress
2026-06-19 08:20:46.895 UTC [16697] WARNING: there is no transaction in progress
2026-06-19 08:20:46.913 UTC [16697] WARNING: there is no transaction in progress
2026-06-19 08:20:46.925 UTC [16697] WARNING: there is no transaction in progress
2026-06-19 08:20:46.938 UTC [16697] WARNING: there is no transaction in progress
2026-06-19 08:20:46.955 UTC [16697] WARNING: there is no transaction in progress
2026-06-19 08:20:46.968 UTC [16697] WARNING: there is no transaction in progress
2026-06-19 08:20:46.977 UTC [16697] WARNING: there is no transaction in progress
2026-06-19 08:20:46.986 UTC [16697] WARNING: there is no transaction in progress
2026-06-19 08:20:46.995 UTC [16697] WARNING: there is no transaction in progress
2026-06-19 08:20:47,920 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 08:20:47,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:44785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 08:20:48,174 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 08:20:48,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:44785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 08:20:48,449 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 08:20:48,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:44785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 08:20:48,726 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 08:20:48,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:44785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 08:20:48,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:44785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 08:20:49,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:44785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-d3n8.c.gcp-upstream.internal';"
2026-06-19 08:20:49,478 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 08:20:49,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:44785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 08:20:49,749 [I] DB_PATCHES have already been applied
I20260619 08:20:49.755359 4702 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ranger-admin:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskR4iS53/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 08:20:49.755436 4702 mini_ranger.cc:193] Using host: 127.4.151.148
I20260619 08:20:49.758689 4702 mini_ranger.cc:227] Ranger admin URL: http://127.4.151.148:39335
Jun 19, 2026 8:20:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 8:20:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 8:20:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 8:20:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 8:20:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 8:20:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 8:20:50 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 08:20:50.915882 4702 mini_ranger.cc:160] Time spent starting Ranger: real 9.260s user 0.001s sys 0.008s
Jun 19, 2026 8:21:12 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 8:21:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 19, 2026 8:21:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 8:21:12 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 8:21:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 8:21:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 8:21:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 8:21:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 8:21:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 8:21:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 8:21:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 8:21:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 8:21:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 8:21:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 8:21:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 8:21:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 8:21:12 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 8:21:13 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 08:21:13.944262 4702 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.151.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.151.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.151.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/rangerkms_127.4.151.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.151.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/rangerkms_127.4.151.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.151.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.151.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.151.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/HTTP_127.4.151.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.151.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/HTTP_127.4.151.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 08:21:14 dist-test-slave-d3n8 krb5kdc[16484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857274, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 08:21:14.036689 4702 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 08:21:14.046461 4702 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 08:21:15.116408 4702 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 08:21:15.116508 4702 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ranger-kms
I20260619 08:21:15.116530 4702 mini_ranger_kms.cc:227] Using postgres at 127.4.151.148:44785
2026-06-19 08:21:15,283 [I] DB FLAVOR :POSTGRES
2026-06-19 08:21:15,283 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 08:21:15,283 [I] Checking connection
2026-06-19 08:21:15,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:44785/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 08:21:15,536 [I] connection success
2026-06-19 08:21:15,537 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 08:21:15,537 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 08:21:15,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:44785/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 08:21:15,785 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 08:21:15,786 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 08:21:15,786 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 08:21:15,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.151.148:44785/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 08:21:16,070 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 08:21:16.579993 4702 mini_ranger_kms.cc:310] Created kms service
I20260619 08:21:16.753647 4702 mini_ranger_kms.cc:326] Created kudu user
I20260619 08:21:16.802356 4702 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 08:21:16 dist-test-slave-d3n8 krb5kdc[16484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857274, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.151.148@KRBTEST.COM
I20260619 08:21:17.225994 4702 mini_ranger_kms.cc:384] Added ranger policy
I20260619 08:21:17.226223 4702 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ranger-kms:/tmp/dist-test-taskR4iS53/build/release/bin/postgresql.jar:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskR4iS53/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 08:21:17.226253 4702 mini_ranger_kms.cc:240] Using host: 127.4.151.148
I20260619 08:21:17.229729 4702 mini_ranger_kms.cc:276] Ranger KMS PID: 16878
I20260619 08:21:17.229825 4702 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.151.148:46351
Jun 19, 2026 8:21:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 8:21:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 8:21:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 8:21:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 8:21:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 8:21:18 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 08:21:19.529764 4702 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.493s user 0.004s sys 0.145s
I20260619 08:21:19.529870 4702 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 08:21:19.529917 4702 mini_ranger_kms.cc:401] 127.4.151.148:46351/kms/v1/keys
Jun 19 08:21:19 dist-test-slave-d3n8 krb5kdc[16484](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781857279, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.151.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 08:21:20 dist-test-slave-d3n8 krb5kdc[16484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857279, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.151.148@KRBTEST.COM for HTTP/127.4.151.148@KRBTEST.COM
Jun 19 08:21:20 dist-test-slave-d3n8 krb5kdc[16484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857279, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.151.148@KRBTEST.COM for HTTP/127.4.151.148@KRBTEST.COM
Jun 19 08:21:22 dist-test-slave-d3n8 krb5kdc[16484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:21:22.849908 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:44341
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:44341
--ranger_config_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.151.148:46351/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:21:22.942899 16928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:22.943183 16928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:22.943228 16928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:22.944470 16928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:21:22.944546 16928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:22.944571 16928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:22.944589 16928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:21:22.944607 16928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:21:22.946305 16928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.151.148:46351/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:44341
--ranger_config_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:44341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:22.946607 16928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:22.946851 16928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:22.953941 16936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:22.953945 16933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:22.953962 16934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:22.953977 16928 server_base.cc:1034] running on GCE node
I20260619 08:21:22.954607 16928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:22.954947 16928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:22.956120 16928 hybrid_clock.cc:648] HybridClock initialized: now 1781857282956088 us; error 50 us; skew 500 ppm
Jun 19 08:21:22 dist-test-slave-d3n8 krb5kdc[16484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857282, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:22.959393 16928 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:21:22.959939 16928 webserver.cc:460] Webserver started at http://127.4.151.190:41929/ using document root <none> and password file <none>
I20260619 08:21:22.960219 16928 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:22.960285 16928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:21:22.960444 16928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 08:21:22 dist-test-slave-d3n8 krb5kdc[16484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857282, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for HTTP/127.4.151.148@KRBTEST.COM
I20260619 08:21:23.015679 16928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "8d14fa17612c4563aa57cf05be686573"
format_stamp: "Formatted at 2026-06-19 08:21:23 on dist-test-slave-d3n8"
server_key: "23db2f4f6da58c2f2026c0762d1eec39"
server_key_iv: "de5d3f3f0b58abffdd0d3f9c22d196a7"
server_key_version: "kuduclusterkey@0"
I20260619 08:21:23.016217 16928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "8d14fa17612c4563aa57cf05be686573"
format_stamp: "Formatted at 2026-06-19 08:21:23 on dist-test-slave-d3n8"
server_key: "23db2f4f6da58c2f2026c0762d1eec39"
server_key_iv: "de5d3f3f0b58abffdd0d3f9c22d196a7"
server_key_version: "kuduclusterkey@0"
I20260619 08:21:23.017856 16928 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:21:23.033087 16943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:23.033428 16928 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:21:23.033605 16928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "8d14fa17612c4563aa57cf05be686573"
format_stamp: "Formatted at 2026-06-19 08:21:23 on dist-test-slave-d3n8"
server_key: "23db2f4f6da58c2f2026c0762d1eec39"
server_key_iv: "de5d3f3f0b58abffdd0d3f9c22d196a7"
server_key_version: "kuduclusterkey@0"
I20260619 08:21:23.033828 16928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:23.052654 16928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:23.055284 16928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:23.055449 16928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:23.060704 16928 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:44341
I20260619 08:21:23.060825 16995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:44341 every 8 connection(s)
I20260619 08:21:23.061249 16928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:21:23.062038 16996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:21:23.064440 16996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573: Bootstrap starting.
I20260619 08:21:23.065074 16996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573: Neither blocks nor log segments found. Creating new log.
I20260619 08:21:23.065447 16996 log.cc:824] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573: Log is configured to *not* fsync() on all Append() calls
I20260619 08:21:23.066336 16996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573: No bootstrap required, opened a new log
I20260619 08:21:23.067241 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 16928
I20260619 08:21:23.067343 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:21:23.067518 4702 external_mini_cluster.cc:1428] Setting key 09f10565478fa6050a0cea5c0734c613
I20260619 08:21:23.069231 16996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d14fa17612c4563aa57cf05be686573" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44341 } }
I20260619 08:21:23.069474 16996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:21:23.069545 16996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d14fa17612c4563aa57cf05be686573, State: Initialized, Role: FOLLOWER
I20260619 08:21:23.069695 16996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d14fa17612c4563aa57cf05be686573" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44341 } }
I20260619 08:21:23.069814 16996 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:21:23.069870 16996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:21:23.069953 16996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:21:23.070816 16996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d14fa17612c4563aa57cf05be686573" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44341 } }
I20260619 08:21:23.070931 16996 leader_election.cc:302] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d14fa17612c4563aa57cf05be686573; no voters:
I20260619 08:21:23.071146 16996 leader_election.cc:288] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:21:23.071239 17001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:21:23.071452 17001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [term 1 LEADER]: Becoming Leader. State: Replica: 8d14fa17612c4563aa57cf05be686573, State: Running, Role: LEADER
I20260619 08:21:23.071444 16996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:21:23.071585 17001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d14fa17612c4563aa57cf05be686573" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44341 } }
I20260619 08:21:23.071877 17001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d14fa17612c4563aa57cf05be686573" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d14fa17612c4563aa57cf05be686573" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44341 } } }
I20260619 08:21:23.071943 17001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [sys.catalog]: This master's current role is: LEADER
I20260619 08:21:23.072073 17001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d14fa17612c4563aa57cf05be686573. Latest consensus state: current_term: 1 leader_uuid: "8d14fa17612c4563aa57cf05be686573" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d14fa17612c4563aa57cf05be686573" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44341 } } }
I20260619 08:21:23.072109 17001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [sys.catalog]: This master's current role is: LEADER
I20260619 08:21:23.072363 17007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:21:23.072880 17007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:21:23.073166 16996 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Jun 19 08:21:23 dist-test-slave-d3n8 krb5kdc[16484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:21:23.083536 17000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:23.068248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60632 (local address 127.4.151.190:44341)
0619 08:21:23.068459 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:23.068470 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:23.068505 (+ 35us) server_negotiation.cc:407] Connection header received
0619 08:21:23.069185 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:23.069195 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:23.069264 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:23.069443 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:23.070254 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:23.071731 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:23.072939 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:23.073609 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:23.073740 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:23.076426 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:23.076430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:23.076430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:23.076443 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:23.078161 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:23.078649 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:23.078653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:23.078653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:23.081126 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:23.081488 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:23.081489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:23.081489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:23.082234 (+ 745us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:23.082435 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:23.082970 (+ 535us) server_negotiation.cc:299] Negotiation successful
0619 08:21:23.083192 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
I20260619 08:21:23.086001 17007 catalog_manager.cc:1347] Generated new cluster ID: 2b4954a066bf42cfbf545ad6b44d46f3
I20260619 08:21:23.086071 17007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:21:23.097486 17007 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:21:23.098392 17007 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:21:23.110332 17007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573: Generated new TSK 0
I20260619 08:21:23.110672 17007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.151.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
I20260619 08:21:23.838109 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.151.148:46351/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:44341
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:21:23.935185 17041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:23.935467 17041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:23.935510 17041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:23.936779 17041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:23.936864 17041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:23.936910 17041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:21:23.938789 17041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.151.148:46351/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:44341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:23.939095 17041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:23.939415 17041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:23.945495 17046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:23.945506 17049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:23.945506 17047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:23.945645 17041 server_base.cc:1034] running on GCE node
I20260619 08:21:23.945971 17041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:23.946259 17041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:23.947487 17041 hybrid_clock.cc:648] HybridClock initialized: now 1781857283947464 us; error 32 us; skew 500 ppm
Jun 19 08:21:23 dist-test-slave-d3n8 krb5kdc[16484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857283, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:23.950484 17041 init.cc:377] Logged in from keytab as kudu/127.4.151.129@KRBTEST.COM (short username kudu)
I20260619 08:21:23.950973 17041 webserver.cc:460] Webserver started at http://127.4.151.129:38931/ using document root <none> and password file <none>
I20260619 08:21:23.951251 17041 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:23.951328 17041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:21:23.951500 17041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 08:21:23 dist-test-slave-d3n8 krb5kdc[16484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857283, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for HTTP/127.4.151.148@KRBTEST.COM
I20260619 08:21:23.968552 17041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "35daca58769742dcb95828a1f3354c8d"
format_stamp: "Formatted at 2026-06-19 08:21:23 on dist-test-slave-d3n8"
server_key: "5dac75e29a88506c1b953281e19e457f"
server_key_iv: "37b8ebddebf719d8b8a54eacaf02b477"
server_key_version: "kuduclusterkey@0"
I20260619 08:21:23.969157 17041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "35daca58769742dcb95828a1f3354c8d"
format_stamp: "Formatted at 2026-06-19 08:21:23 on dist-test-slave-d3n8"
server_key: "5dac75e29a88506c1b953281e19e457f"
server_key_iv: "37b8ebddebf719d8b8a54eacaf02b477"
server_key_version: "kuduclusterkey@0"
I20260619 08:21:23.972055 17041 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260619 08:21:23.987035 17056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:23.987270 17041 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:21:23.987363 17041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "35daca58769742dcb95828a1f3354c8d"
format_stamp: "Formatted at 2026-06-19 08:21:23 on dist-test-slave-d3n8"
server_key: "5dac75e29a88506c1b953281e19e457f"
server_key_iv: "37b8ebddebf719d8b8a54eacaf02b477"
server_key_version: "kuduclusterkey@0"
I20260619 08:21:23.987470 17041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:24.005455 17041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:24.007531 17041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:24.007690 17041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:24.007953 17041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:21:24.008347 17041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:21:24.008407 17041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:24.008442 17041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:21:24.008469 17041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:24.015826 17041 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:45871
I20260619 08:21:24.016376 17041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
I20260619 08:21:24.017124 17169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:45871 every 8 connection(s)
I20260619 08:21:24.018420 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 17041
I20260619 08:21:24.018530 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:21:24.018705 4702 external_mini_cluster.cc:1428] Setting key 77865fc8b0a27a4631bf18abcbb46f55
Jun 19 08:21:24 dist-test-slave-d3n8 krb5kdc[16484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857283, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.151.130@KRBTEST.COM; defaulting to no policy
I20260619 08:21:24.033252 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:24.017561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:56991 (local address 127.4.151.190:44341)
0619 08:21:24.017707 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:24.017709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:24.017754 (+ 45us) server_negotiation.cc:407] Connection header received
0619 08:21:24.018449 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:24.018450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:24.018466 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:24.018555 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:24.019385 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:24.020144 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:24.021682 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:24.021970 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:24.022044 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:24.026760 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:24.026762 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:24.026762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:24.026771 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:24.029692 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:24.032031 (+ 2339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:24.032033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:24.032034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:24.032118 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:24.032416 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:24.032417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:24.032417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:24.032585 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:24.032661 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:24.033032 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 08:21:24.033133 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":95,"threads_started":1}
I20260619 08:21:24.033381 17172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:24.017260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44341 (local address 127.4.151.129:56991)
0619 08:21:24.017598 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:24.017663 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:24.018322 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:24.018675 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:24.018680 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:24.018944 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:24.019269 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:24.019275 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:24.020257 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:24.020258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:24.021566 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:24.021570 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:24.022136 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:24.022138 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:24.022319 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:24.024052 (+ 1733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:24.024063 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:24.026636 (+ 2573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:24.029812 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:24.029822 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:24.029825 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:24.031890 (+ 2065us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:24.032204 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:24.032205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:24.032205 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:24.032320 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:24.032663 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:24.032670 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:24.032857 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 08:21:24.032946 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 08:21:24.033079 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":84,"threads_started":1}
I20260619 08:21:24.033939 17170 heartbeater.cc:347] Connected to a master server at 127.4.151.190:44341
I20260619 08:21:24.034067 17170 heartbeater.cc:464] Registering TS with master...
I20260619 08:21:24.034286 17170 heartbeater.cc:511] Master 127.4.151.190:44341 requested a full tablet report, sending...
I20260619 08:21:24.034780 16960 ts_manager.cc:194] Registered new tserver with Master: 35daca58769742dcb95828a1f3354c8d (127.4.151.129:45871)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.130@KRBTEST.COM" created.
I20260619 08:21:24.035715 16960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.129@KRBTEST.COM'} at 127.4.151.129:56991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
I20260619 08:21:24.053195 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.151.148:46351/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:44341
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:21:24.171749 17178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:24.172071 17178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:24.172119 17178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:24.174131 17178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:24.174224 17178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:24.174269 17178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:21:24.176559 17178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.151.148:46351/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:44341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:24.176841 17178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:24.177104 17178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:24.184487 17188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:24.184808 17186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:24.185135 17185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:24.185081 17178 server_base.cc:1034] running on GCE node
I20260619 08:21:24.185516 17178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:24.185793 17178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:24.186961 17178 hybrid_clock.cc:648] HybridClock initialized: now 1781857284186939 us; error 36 us; skew 500 ppm
Jun 19 08:21:24 dist-test-slave-d3n8 krb5kdc[16484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857284, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:24.190353 17178 init.cc:377] Logged in from keytab as kudu/127.4.151.130@KRBTEST.COM (short username kudu)
I20260619 08:21:24.191039 17178 webserver.cc:460] Webserver started at http://127.4.151.130:42235/ using document root <none> and password file <none>
I20260619 08:21:24.191366 17178 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:24.191481 17178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:21:24.191723 17178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 08:21:24 dist-test-slave-d3n8 krb5kdc[16484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857284, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for HTTP/127.4.151.148@KRBTEST.COM
Jun 19 08:21:24 dist-test-slave-d3n8 krb5kdc[16484](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781857284, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:24.206094 17178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "fbe3f9e99ffb41f2967e2471c010407f"
format_stamp: "Formatted at 2026-06-19 08:21:24 on dist-test-slave-d3n8"
server_key: "9fd0cb6c1408104820b500f708e33f77"
server_key_iv: "9917d09f76beceb3380b8e38dacec641"
server_key_version: "kuduclusterkey@0"
I20260619 08:21:24.206821 17178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "fbe3f9e99ffb41f2967e2471c010407f"
format_stamp: "Formatted at 2026-06-19 08:21:24 on dist-test-slave-d3n8"
server_key: "9fd0cb6c1408104820b500f708e33f77"
server_key_iv: "9917d09f76beceb3380b8e38dacec641"
server_key_version: "kuduclusterkey@0"
I20260619 08:21:24.217940 17178 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.002s sys 0.001s
I20260619 08:21:24.240355 17195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:24.240654 17178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 08:21:24.240775 17178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "fbe3f9e99ffb41f2967e2471c010407f"
format_stamp: "Formatted at 2026-06-19 08:21:24 on dist-test-slave-d3n8"
server_key: "9fd0cb6c1408104820b500f708e33f77"
server_key_iv: "9917d09f76beceb3380b8e38dacec641"
server_key_version: "kuduclusterkey@0"
I20260619 08:21:24.240893 17178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:24.274269 17178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:24.277276 17178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:24.277473 17178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:24.277736 17178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:21:24.278280 17178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:21:24.278345 17178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:24.278378 17178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:21:24.278398 17178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:24.286314 17178 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:37131
I20260619 08:21:24.286760 17178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
I20260619 08:21:24.286890 17309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:37131 every 8 connection(s)
Jun 19 08:21:24 dist-test-slave-d3n8 krb5kdc[16484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857284, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:21:24.296710 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 17178
I20260619 08:21:24.296813 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:21:24.297001 4702 external_mini_cluster.cc:1428] Setting key b5fae1463e223a620a9f2add22c9155d
I20260619 08:21:24.299427 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:24.287612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:48161 (local address 127.4.151.190:44341)
0619 08:21:24.287847 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:24.287849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:24.287994 (+ 145us) server_negotiation.cc:407] Connection header received
0619 08:21:24.288704 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:24.288705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:24.288721 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:24.288795 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:24.289607 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:24.290412 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:24.292083 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:24.292440 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:24.292521 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:24.295366 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:24.295368 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:24.295369 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:24.295378 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:24.297250 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:24.298174 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:24.298176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:24.298176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:24.298249 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:24.298579 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:24.298580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:24.298580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:24.298747 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:24.298806 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:24.299204 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 08:21:24.299315 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260619 08:21:24.300074 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:24.287590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44341 (local address 127.4.151.130:48161)
0619 08:21:24.287823 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:24.287876 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:24.288551 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:24.288876 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:24.288880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:24.289158 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:24.289493 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:24.289500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:24.290539 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:24.290541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:24.292001 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:24.292004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:24.292556 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:24.292557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:24.292731 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:24.293465 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:24.293477 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:24.295250 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:24.297677 (+ 2427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:24.297687 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:24.297690 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:24.298069 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:24.298331 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:24.298331 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:24.298332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:24.298496 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:24.298842 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:24.298849 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:24.299041 (+ 192us) client_negotiation.cc:769] Sending connection context
0619 08:21:24.299566 (+ 525us) client_negotiation.cc:240] Negotiation successful
0619 08:21:24.299798 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":95,"threads_started":1}
I20260619 08:21:24.300549 17310 heartbeater.cc:347] Connected to a master server at 127.4.151.190:44341
I20260619 08:21:24.300643 17310 heartbeater.cc:464] Registering TS with master...
I20260619 08:21:24.300864 17310 heartbeater.cc:511] Master 127.4.151.190:44341 requested a full tablet report, sending...
I20260619 08:21:24.301252 16960 ts_manager.cc:194] Registered new tserver with Master: fbe3f9e99ffb41f2967e2471c010407f (127.4.151.130:37131)
I20260619 08:21:24.301841 16960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.130@KRBTEST.COM'} at 127.4.151.130:48161
WARNING: no policy specified for kudu/127.4.151.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
I20260619 08:21:24.338922 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.151.148:46351/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:44341
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:21:24.475320 17315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:24.475641 17315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:24.475682 17315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:24.477809 17315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:24.477892 17315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:24.477943 17315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:21:24.480499 17315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.151.148:46351/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:44341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:24.480798 17315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:24.481097 17315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:24.488766 17321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:24.488780 17320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:24.489059 17323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:24.490134 17315 server_base.cc:1034] running on GCE node
I20260619 08:21:24.490412 17315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:24.490733 17315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:24.491905 17315 hybrid_clock.cc:648] HybridClock initialized: now 1781857284491881 us; error 44 us; skew 500 ppm
Jun 19 08:21:24 dist-test-slave-d3n8 krb5kdc[16484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857284, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:24.495457 17315 init.cc:377] Logged in from keytab as kudu/127.4.151.131@KRBTEST.COM (short username kudu)
I20260619 08:21:24.496065 17315 webserver.cc:460] Webserver started at http://127.4.151.131:38309/ using document root <none> and password file <none>
I20260619 08:21:24.496363 17315 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:24.496433 17315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:21:24.496582 17315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 08:21:24 dist-test-slave-d3n8 krb5kdc[16484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857284, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for HTTP/127.4.151.148@KRBTEST.COM
I20260619 08:21:24.510804 17315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "2f3c6fe7804e4c4b8c500f510401139e"
format_stamp: "Formatted at 2026-06-19 08:21:24 on dist-test-slave-d3n8"
server_key: "650da66513f14c866b17b03a9b82578d"
server_key_iv: "8147da7e97f7f99a46b9c97af3bcd9c9"
server_key_version: "kuduclusterkey@0"
I20260619 08:21:24.511313 17315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "2f3c6fe7804e4c4b8c500f510401139e"
format_stamp: "Formatted at 2026-06-19 08:21:24 on dist-test-slave-d3n8"
server_key: "650da66513f14c866b17b03a9b82578d"
server_key_iv: "8147da7e97f7f99a46b9c97af3bcd9c9"
server_key_version: "kuduclusterkey@0"
I20260619 08:21:24.518224 17315 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.000s
I20260619 08:21:24.549713 17330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:24.550222 17315 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.000s sys 0.000s
I20260619 08:21:24.550387 17315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "2f3c6fe7804e4c4b8c500f510401139e"
format_stamp: "Formatted at 2026-06-19 08:21:24 on dist-test-slave-d3n8"
server_key: "650da66513f14c866b17b03a9b82578d"
server_key_iv: "8147da7e97f7f99a46b9c97af3bcd9c9"
server_key_version: "kuduclusterkey@0"
I20260619 08:21:24.550541 17315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:24.570374 17315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:24.574142 17315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:24.574395 17315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:24.574635 17315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:21:24.574994 17315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:21:24.575048 17315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:24.575098 17315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:21:24.575135 17315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:24.581559 17315 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:33781
I20260619 08:21:24.582024 17315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
I20260619 08:21:24.584560 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 17315
I20260619 08:21:24.584669 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:21:24.584884 4702 external_mini_cluster.cc:1428] Setting key 4f278c4f39db66ac413d9a10b1a87da7
I20260619 08:21:24.585307 17443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:33781 every 8 connection(s)
Jun 19 08:21:24 dist-test-slave-d3n8 krb5kdc[16484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857284, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:21:24.597273 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:24.585978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:56131 (local address 127.4.151.190:44341)
0619 08:21:24.586025 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:24.586027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:24.586537 (+ 510us) server_negotiation.cc:407] Connection header received
0619 08:21:24.587257 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:24.587258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:24.587270 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:24.587301 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:24.588158 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:24.588876 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:24.590470 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:24.590757 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:24.590809 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:24.593664 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:24.593666 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:24.593667 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:24.593676 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:24.595519 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:24.596109 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:24.596111 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:24.596111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:24.596178 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:24.596470 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:24.596471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:24.596471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:24.596617 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:24.596666 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:24.597085 (+ 419us) server_negotiation.cc:299] Negotiation successful
0619 08:21:24.597176 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:21:24.598142 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:24.586003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44341 (local address 127.4.151.131:56131)
0619 08:21:24.586405 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:24.586452 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:24.587130 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:24.587494 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:24.587499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:24.587755 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:24.588034 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:24.588052 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:24.589009 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:24.589010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:24.590359 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:24.590362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:24.590936 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:24.590937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:24.591111 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:24.591757 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:24.591768 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:24.593542 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:24.595659 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:24.595668 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:24.595671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:24.595995 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:24.596254 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:24.596255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:24.596256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:24.596392 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:24.596720 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:24.596727 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:24.596903 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 08:21:24.597701 (+ 798us) client_negotiation.cc:240] Negotiation successful
0619 08:21:24.597883 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":95,"threads_started":1}
I20260619 08:21:24.598805 17444 heartbeater.cc:347] Connected to a master server at 127.4.151.190:44341
I20260619 08:21:24.598896 17444 heartbeater.cc:464] Registering TS with master...
I20260619 08:21:24.599092 17444 heartbeater.cc:511] Master 127.4.151.190:44341 requested a full tablet report, sending...
I20260619 08:21:24.599493 16960 ts_manager.cc:194] Registered new tserver with Master: 2f3c6fe7804e4c4b8c500f510401139e (127.4.151.131:33781)
I20260619 08:21:24.599972 16960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.131@KRBTEST.COM'} at 127.4.151.131:56131
I20260619 08:21:24.607466 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:21:24.608417 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 17041
I20260619 08:21:24.615541 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 17178
I20260619 08:21:24.621737 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 17315
I20260619 08:21:24.629189 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 16928
I20260619 08:21:24.636248 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:44341
--webserver_interface=127.4.151.190
--webserver_port=41929
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:44341
--ranger_config_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.151.148:46351/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.151.148:46351/kms with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:21:24.727895 17447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:24.728190 17447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:24.728232 17447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:24.729622 17447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:21:24.729694 17447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:24.729719 17447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:24.729748 17447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:21:24.729779 17447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:21:24.731508 17447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.151.148:46351/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:44341
--ranger_config_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:44341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=41929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:24.731813 17447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:24.732105 17447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:24.737694 17455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:24.738278 17453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:24.738332 17452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:24.738831 17447 server_base.cc:1034] running on GCE node
I20260619 08:21:24.739205 17447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:24.739537 17447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:24.740840 17447 hybrid_clock.cc:648] HybridClock initialized: now 1781857284740805 us; error 46 us; skew 500 ppm
Jun 19 08:21:24 dist-test-slave-d3n8 krb5kdc[16484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857284, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:24.743903 17447 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:21:24.744488 17447 webserver.cc:460] Webserver started at http://127.4.151.190:41929/ using document root <none> and password file <none>
I20260619 08:21:24.744783 17447 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:24.744872 17447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 08:21:24 dist-test-slave-d3n8 krb5kdc[16484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857284, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for HTTP/127.4.151.148@KRBTEST.COM
I20260619 08:21:24.830783 17447 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:21:24.832074 17463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:24.832329 17447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:21:24.832443 17447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "8d14fa17612c4563aa57cf05be686573"
format_stamp: "Formatted at 2026-06-19 08:21:23 on dist-test-slave-d3n8"
server_key: "23db2f4f6da58c2f2026c0762d1eec39"
server_key_iv: "de5d3f3f0b58abffdd0d3f9c22d196a7"
server_key_version: "kuduclusterkey@0"
I20260619 08:21:24.832957 17447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:24.868562 17447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:24.871166 17447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:24.871347 17447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:24.877032 17447 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:44341
I20260619 08:21:24.877135 17515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:44341 every 8 connection(s)
I20260619 08:21:24.877632 17447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:21:24.879225 17516 sys_catalog.cc:263] Verifying existing consensus state
I20260619 08:21:24.879962 17516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573: Bootstrap starting.
I20260619 08:21:24.885071 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 17447
I20260619 08:21:24.885445 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.151.148:46351/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:45871
--local_ip_for_outbound_sockets=127.4.151.129
--tserver_master_addrs=127.4.151.190:44341
--webserver_port=38931
--webserver_interface=127.4.151.129
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.151.148:46351/kms with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:21:24.890565 17516 log.cc:824] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573: Log is configured to *not* fsync() on all Append() calls
I20260619 08:21:24.892499 17516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:21:24.892851 17516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573: Bootstrap complete.
I20260619 08:21:24.895630 17516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d14fa17612c4563aa57cf05be686573" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44341 } }
I20260619 08:21:24.895931 17516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d14fa17612c4563aa57cf05be686573, State: Initialized, Role: FOLLOWER
I20260619 08:21:24.896145 17516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d14fa17612c4563aa57cf05be686573" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44341 } }
I20260619 08:21:24.896358 17516 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:21:24.896492 17516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:21:24.896567 17516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [term 1 FOLLOWER]: Advancing to term 2
I20260619 08:21:24.897990 17516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d14fa17612c4563aa57cf05be686573" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44341 } }
I20260619 08:21:24.898140 17516 leader_election.cc:302] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d14fa17612c4563aa57cf05be686573; no voters:
I20260619 08:21:24.898422 17516 leader_election.cc:288] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 08:21:24.898468 17520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 08:21:24.898715 17520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [term 2 LEADER]: Becoming Leader. State: Replica: 8d14fa17612c4563aa57cf05be686573, State: Running, Role: LEADER
I20260619 08:21:24.898873 17520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d14fa17612c4563aa57cf05be686573" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44341 } }
I20260619 08:21:24.898938 17516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:21:24.899734 17520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8d14fa17612c4563aa57cf05be686573" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d14fa17612c4563aa57cf05be686573" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44341 } } }
I20260619 08:21:24.899878 17520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [sys.catalog]: This master's current role is: LEADER
I20260619 08:21:24.900524 17520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d14fa17612c4563aa57cf05be686573. Latest consensus state: current_term: 2 leader_uuid: "8d14fa17612c4563aa57cf05be686573" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d14fa17612c4563aa57cf05be686573" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44341 } } }
I20260619 08:21:24.900652 17520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573 [sys.catalog]: This master's current role is: LEADER
I20260619 08:21:24.901473 17524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:21:24.901882 17516 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 08:21:24.902088 17524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:21:24.902225 17524 catalog_manager.cc:1259] Loaded cluster ID: 2b4954a066bf42cfbf545ad6b44d46f3
I20260619 08:21:24.902266 17524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:21:24.903221 17524 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:21:24.903450 17524 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573: Loaded TSK: 0
I20260619 08:21:24.903704 17524 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 08:21:25.016731 17518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:25.017117 17518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:25.017200 17518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:25.019296 17518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:25.019423 17518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:25.019505 17518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:21:25.022014 17518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.151.148:46351/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:45871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=38931
--tserver_master_addrs=127.4.151.190:44341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:25.022543 17518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:25.022914 17518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:25.030352 17554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:25.030369 17553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:25.031543 17556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:25.032042 17518 server_base.cc:1034] running on GCE node
I20260619 08:21:25.032300 17518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:25.032631 17518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:25.034309 17518 hybrid_clock.cc:648] HybridClock initialized: now 1781857285034273 us; error 48 us; skew 500 ppm
Jun 19 08:21:25 dist-test-slave-d3n8 krb5kdc[16484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857285, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:25.039808 17518 init.cc:377] Logged in from keytab as kudu/127.4.151.129@KRBTEST.COM (short username kudu)
I20260619 08:21:25.040759 17518 webserver.cc:460] Webserver started at http://127.4.151.129:38931/ using document root <none> and password file <none>
I20260619 08:21:25.041121 17518 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:25.041242 17518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 08:21:25 dist-test-slave-d3n8 krb5kdc[16484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857285, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for HTTP/127.4.151.148@KRBTEST.COM
I20260619 08:21:25.075680 17518 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260619 08:21:25.076894 17564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:25.077093 17518 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:21:25.077190 17518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "35daca58769742dcb95828a1f3354c8d"
format_stamp: "Formatted at 2026-06-19 08:21:23 on dist-test-slave-d3n8"
server_key: "5dac75e29a88506c1b953281e19e457f"
server_key_iv: "37b8ebddebf719d8b8a54eacaf02b477"
server_key_version: "kuduclusterkey@0"
I20260619 08:21:25.077760 17518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:25.127079 17518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:25.129448 17518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:25.129634 17518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:25.129894 17518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:21:25.130306 17518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:21:25.130362 17518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:25.130403 17518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:21:25.130429 17518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:25.139097 17518 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:45871
I20260619 08:21:25.139634 17518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
I20260619 08:21:25.145108 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 17518
I20260619 08:21:25.145464 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.151.148:46351/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:37131
--local_ip_for_outbound_sockets=127.4.151.130
--tserver_master_addrs=127.4.151.190:44341
--webserver_port=42235
--webserver_interface=127.4.151.130
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.151.148:46351/kms with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:21:25.146762 17677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:45871 every 8 connection(s)
Jun 19 08:21:25 dist-test-slave-d3n8 krb5kdc[16484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857285, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:21:25.161513 17680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:25.145992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44341 (local address 127.4.151.129:39639)
0619 08:21:25.146933 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:25.146986 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:25.147710 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:25.148424 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:25.148428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:25.148707 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:25.149087 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:25.149095 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:25.151287 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:25.151289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:25.152865 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:25.152872 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:25.154099 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:25.154101 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:25.154264 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:25.154969 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:25.154982 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:25.156956 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:25.159629 (+ 2673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:25.159637 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:25.159639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:25.159956 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:25.160224 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:25.160224 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:25.160225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:25.160346 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:25.160825 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:25.160830 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:25.160976 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 08:21:25.161082 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 08:21:25.161247 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"thread_start_us":137,"threads_started":1}
I20260619 08:21:25.162791 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:25.146446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:39639 (local address 127.4.151.190:44341)
0619 08:21:25.147378 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:25.147389 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:25.147435 (+ 46us) server_negotiation.cc:407] Connection header received
0619 08:21:25.148036 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:25.148042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:25.148107 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:25.148238 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:25.149510 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:25.150995 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:25.153013 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:25.153808 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:25.153905 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:25.157119 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:25.157125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:25.157125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:25.157140 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:25.159518 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:25.160046 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:25.160049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:25.160050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:25.160154 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:25.160413 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:25.160414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:25.160414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:25.160724 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:25.160797 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:25.162320 (+ 1523us) server_negotiation.cc:299] Negotiation successful
0619 08:21:25.162481 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":94,"threads_started":1}
I20260619 08:21:25.163898 17678 heartbeater.cc:347] Connected to a master server at 127.4.151.190:44341
I20260619 08:21:25.164029 17678 heartbeater.cc:464] Registering TS with master...
I20260619 08:21:25.164230 17678 heartbeater.cc:511] Master 127.4.151.190:44341 requested a full tablet report, sending...
W20260619 08:21:25.165136 17678 heartbeater.cc:644] Failed to heartbeat to 127.4.151.190:44341 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 08:21:25.165232 17678 heartbeater.cc:464] Registering TS with master...
I20260619 08:21:25.165298 17678 heartbeater.cc:511] Master 127.4.151.190:44341 requested a full tablet report, sending...
I20260619 08:21:25.165630 17678 heartbeater.cc:464] Registering TS with master...
I20260619 08:21:25.165697 17678 heartbeater.cc:511] Master 127.4.151.190:44341 requested a full tablet report, sending...
W20260619 08:21:25.165951 17678 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 08:21:25.273501 17682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:25.273833 17682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:25.273888 17682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:25.275827 17682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:25.275900 17682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:25.275950 17682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:21:25.278475 17682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.151.148:46351/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:37131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=42235
--tserver_master_addrs=127.4.151.190:44341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:25.278780 17682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:25.279078 17682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:25.288381 17687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:25.288386 17688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:25.288370 17690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:25.289276 17682 server_base.cc:1034] running on GCE node
I20260619 08:21:25.289554 17682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:25.289878 17682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:25.291038 17682 hybrid_clock.cc:648] HybridClock initialized: now 1781857285291017 us; error 35 us; skew 500 ppm
Jun 19 08:21:25 dist-test-slave-d3n8 krb5kdc[16484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857285, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:25.294885 17682 init.cc:377] Logged in from keytab as kudu/127.4.151.130@KRBTEST.COM (short username kudu)
I20260619 08:21:25.295455 17682 webserver.cc:460] Webserver started at http://127.4.151.130:42235/ using document root <none> and password file <none>
I20260619 08:21:25.295712 17682 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:25.295776 17682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 08:21:25 dist-test-slave-d3n8 krb5kdc[16484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857285, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for HTTP/127.4.151.148@KRBTEST.COM
I20260619 08:21:25.315182 17682 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:21:25.316310 17698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:25.316522 17682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 08:21:25.316605 17682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "fbe3f9e99ffb41f2967e2471c010407f"
format_stamp: "Formatted at 2026-06-19 08:21:24 on dist-test-slave-d3n8"
server_key: "9fd0cb6c1408104820b500f708e33f77"
server_key_iv: "9917d09f76beceb3380b8e38dacec641"
server_key_version: "kuduclusterkey@0"
I20260619 08:21:25.317056 17682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:25.344748 17682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:25.347822 17682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:25.348037 17682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:25.348332 17682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:21:25.348759 17682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:21:25.348800 17682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:25.348824 17682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:21:25.348837 17682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:25.356874 17682 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:37131
I20260619 08:21:25.357714 17811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:37131 every 8 connection(s)
I20260619 08:21:25.357920 17682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
I20260619 08:21:25.364178 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 17682
I20260619 08:21:25.364496 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.151.148:46351/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:33781
--local_ip_for_outbound_sockets=127.4.151.131
--tserver_master_addrs=127.4.151.190:44341
--webserver_port=38309
--webserver_interface=127.4.151.131
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.151.148:46351/kms with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 08:21:25 dist-test-slave-d3n8 krb5kdc[16484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857285, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:21:25.369427 17814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:25.358764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44341 (local address 127.4.151.130:49185)
0619 08:21:25.358988 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:25.359031 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:25.359575 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:25.359841 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:25.359845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:25.360082 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:25.360354 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:25.360359 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:25.361098 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:25.361099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:25.362284 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:25.362287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:25.362748 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:25.362750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:25.362885 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:25.363815 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:25.363824 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:25.365254 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:25.367492 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:25.367501 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:25.367505 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:25.367830 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:25.368102 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:25.368103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:25.368103 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:25.368259 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:25.368579 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:25.368586 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:25.368774 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 08:21:25.368889 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 08:21:25.369077 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":80,"threads_started":1}
I20260619 08:21:25.369938 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:25.358858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:49185 (local address 127.4.151.190:44341)
0619 08:21:25.358918 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:25.358920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:25.359136 (+ 216us) server_negotiation.cc:407] Connection header received
0619 08:21:25.359696 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:25.359698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:25.359713 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:25.359770 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:25.360459 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:25.360979 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:25.362386 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:25.362615 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:25.362716 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:25.365508 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:25.365510 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:25.365510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:25.365518 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:25.367395 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:25.367951 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:25.367952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:25.367953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:25.368022 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:25.368334 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:25.368335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:25.368335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:25.368492 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:25.368552 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:25.369713 (+ 1161us) server_negotiation.cc:299] Negotiation successful
0619 08:21:25.369826 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 08:21:25.370292 17812 heartbeater.cc:347] Connected to a master server at 127.4.151.190:44341
I20260619 08:21:25.370393 17812 heartbeater.cc:464] Registering TS with master...
I20260619 08:21:25.370554 17812 heartbeater.cc:511] Master 127.4.151.190:44341 requested a full tablet report, sending...
W20260619 08:21:25.371217 17812 heartbeater.cc:644] Failed to heartbeat to 127.4.151.190:44341 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 08:21:25.371279 17812 heartbeater.cc:464] Registering TS with master...
I20260619 08:21:25.371330 17812 heartbeater.cc:511] Master 127.4.151.190:44341 requested a full tablet report, sending...
I20260619 08:21:25.371682 17812 heartbeater.cc:464] Registering TS with master...
I20260619 08:21:25.371762 17812 heartbeater.cc:511] Master 127.4.151.190:44341 requested a full tablet report, sending...
W20260619 08:21:25.372104 17812 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 08:21:25.471539 17815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:25.471877 17815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:25.471920 17815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:25.474081 17815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:25.474164 17815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:25.474211 17815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:21:25.476902 17815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.151.148:46351/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:33781
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=38309
--tserver_master_addrs=127.4.151.190:44341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:25.477242 17815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:25.477571 17815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:25.484580 17821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:25.484611 17820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:25.484580 17823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:25.485273 17815 server_base.cc:1034] running on GCE node
I20260619 08:21:25.485566 17815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:25.485888 17815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:25.487087 17815 hybrid_clock.cc:648] HybridClock initialized: now 1781857285487071 us; error 33 us; skew 500 ppm
Jun 19 08:21:25 dist-test-slave-d3n8 krb5kdc[16484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857285, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:25.490676 17815 init.cc:377] Logged in from keytab as kudu/127.4.151.131@KRBTEST.COM (short username kudu)
I20260619 08:21:25.491309 17815 webserver.cc:460] Webserver started at http://127.4.151.131:38309/ using document root <none> and password file <none>
I20260619 08:21:25.491757 17815 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:25.491904 17815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 08:21:25 dist-test-slave-d3n8 krb5kdc[16484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857285, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for HTTP/127.4.151.148@KRBTEST.COM
I20260619 08:21:25.517948 17815 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260619 08:21:25.519222 17831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:25.519469 17815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:21:25.519567 17815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "2f3c6fe7804e4c4b8c500f510401139e"
format_stamp: "Formatted at 2026-06-19 08:21:24 on dist-test-slave-d3n8"
server_key: "650da66513f14c866b17b03a9b82578d"
server_key_iv: "8147da7e97f7f99a46b9c97af3bcd9c9"
server_key_version: "kuduclusterkey@0"
I20260619 08:21:25.520081 17815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:25.540870 17815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:25.543480 17815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:25.543651 17815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:25.543891 17815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:21:25.544313 17815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:21:25.544378 17815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:25.544412 17815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:21:25.544431 17815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:25.551857 17815 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:33781
I20260619 08:21:25.551919 17945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:33781 every 8 connection(s)
I20260619 08:21:25.552393 17815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
I20260619 08:21:25.556084 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 17815
Jun 19 08:21:25 dist-test-slave-d3n8 krb5kdc[16484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857285, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:21:25.570215 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:25.553498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:36329 (local address 127.4.151.190:44341)
0619 08:21:25.553574 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:25.553577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:25.556595 (+ 3018us) server_negotiation.cc:407] Connection header received
0619 08:21:25.557360 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:25.557362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:25.557401 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:25.557436 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:25.558413 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:25.559163 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:25.560345 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:25.560615 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:25.560685 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:25.566299 (+ 5614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:25.566301 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:25.566302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:25.566310 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:25.568231 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:25.568849 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:25.568850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:25.568851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:25.568914 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:25.569233 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:25.569234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:25.569235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:25.569413 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:25.569471 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:25.569913 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 08:21:25.570050 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 08:21:25.570598 17948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:25.553763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44341 (local address 127.4.151.131:36329)
0619 08:21:25.556415 (+ 2652us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:25.556475 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:25.557202 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:25.557650 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:25.557655 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:25.557928 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:25.558280 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:25.558288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:25.559311 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:25.559311 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:25.560226 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:25.560229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:25.560795 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:25.560797 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:25.560954 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:25.564369 (+ 3415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:25.564381 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:25.566162 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:25.568355 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:25.568367 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:25.568370 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:25.568741 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:25.568993 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:25.568994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:25.568994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:25.569143 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:25.569526 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:25.569533 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:25.569727 (+ 194us) client_negotiation.cc:769] Sending connection context
0619 08:21:25.569855 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 08:21:25.570037 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2592,"thread_start_us":128,"threads_started":1}
I20260619 08:21:25.572125 17946 heartbeater.cc:347] Connected to a master server at 127.4.151.190:44341
I20260619 08:21:25.572221 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:25.558861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60634 (local address 127.4.151.190:44341)
0619 08:21:25.560904 (+ 2043us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:25.560906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:25.561061 (+ 155us) server_negotiation.cc:407] Connection header received
0619 08:21:25.561175 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:25.561176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:25.561563 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:25.561617 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:25.562542 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:25.563313 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:25.564448 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:25.564744 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:25.564790 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:25.566451 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:25.566453 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:25.566453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:25.566462 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:25.570205 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:25.571016 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:25.571017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:25.571017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:25.571103 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:25.571557 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:25.571558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:25.571558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:25.571730 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:25.571914 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:25.571999 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 08:21:25.572111 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2014,"thread_start_us":356,"threads_started":1}
I20260619 08:21:25.572245 17946 heartbeater.cc:464] Registering TS with master...
I20260619 08:21:25.572451 17946 heartbeater.cc:511] Master 127.4.151.190:44341 requested a full tablet report, sending...
W20260619 08:21:25.573328 17946 heartbeater.cc:644] Failed to heartbeat to 127.4.151.190:44341 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 08:21:25.573436 17946 heartbeater.cc:464] Registering TS with master...
I20260619 08:21:25.573522 17946 heartbeater.cc:511] Master 127.4.151.190:44341 requested a full tablet report, sending...
I20260619 08:21:25.573845 17946 heartbeater.cc:464] Registering TS with master...
I20260619 08:21:25.573922 17946 heartbeater.cc:511] Master 127.4.151.190:44341 requested a full tablet report, sending...
W20260619 08:21:25.574304 17946 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 08:21:26.166143 17678 heartbeater.cc:464] Registering TS with master...
I20260619 08:21:26.166332 17678 heartbeater.cc:511] Master 127.4.151.190:44341 requested a full tablet report, sending...
I20260619 08:21:26.166752 17479 ts_manager.cc:194] Registered new tserver with Master: 35daca58769742dcb95828a1f3354c8d (127.4.151.129:45871)
I20260619 08:21:26.167795 17479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.129@KRBTEST.COM'} at 127.4.151.129:39639
Jun 19 08:21:26 dist-test-slave-d3n8 krb5kdc[16484](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781857286, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:26.372318 17812 heartbeater.cc:464] Registering TS with master...
I20260619 08:21:26.372490 17812 heartbeater.cc:511] Master 127.4.151.190:44341 requested a full tablet report, sending...
I20260619 08:21:26.372954 17479 ts_manager.cc:194] Registered new tserver with Master: fbe3f9e99ffb41f2967e2471c010407f (127.4.151.130:37131)
I20260619 08:21:26.373519 17479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.130@KRBTEST.COM'} at 127.4.151.130:49185
Jun 19 08:21:26 dist-test-slave-d3n8 krb5kdc[16484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857286, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for HTTP/127.4.151.148@KRBTEST.COM
I20260619 08:21:26.574508 17946 heartbeater.cc:464] Registering TS with master...
I20260619 08:21:26.574676 17946 heartbeater.cc:511] Master 127.4.151.190:44341 requested a full tablet report, sending...
I20260619 08:21:26.575068 17479 ts_manager.cc:194] Registered new tserver with Master: 2f3c6fe7804e4c4b8c500f510401139e (127.4.151.131:33781)
I20260619 08:21:26.575481 17479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.131@KRBTEST.COM'} at 127.4.151.131:36329
I20260619 08:21:26.581522 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:21:26.598166 17968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:26.582524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60650 (local address 127.4.151.190:44341)
0619 08:21:26.582823 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:26.582825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:26.582832 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:21:26.582855 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:26.582856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:26.582868 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:26.582924 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:26.583664 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:26.584442 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:26.585539 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:26.585916 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:26.585985 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:26.594851 (+ 8866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:26.594853 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:26.594854 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:26.594866 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:26.596606 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:26.597146 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:26.597147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:26.597147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:26.597223 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:26.597449 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:26.597450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:26.597450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:26.597597 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:26.597684 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:26.597931 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 08:21:26.598040 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":72,"threads_started":1}
I20260619 08:21:26.600729 17479 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:21:26.603003 17479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:21:26.627898 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:26.617707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:33781 (local address 127.0.0.1:40492)
0619 08:21:26.621467 (+ 3760us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:26.621518 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:26.621594 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:26.623634 (+ 2040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:26.623637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:26.623640 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:21:26.623722 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:26.623724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:26.624978 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:26.624979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:26.626599 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:26.626602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:26.627597 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:26.627598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:26.627676 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:26.627689 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 08:21:26.627753 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 08:21:26.627807 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3719,"thread_start_us":74,"threads_started":1}
I20260619 08:21:26.628486 17976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:26.618433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40492 (local address 127.4.151.131:33781)
0619 08:21:26.623369 (+ 4936us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:26.623374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:26.623385 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:21:26.623418 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:26.623420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:26.623436 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:26.623507 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:21:26.623798 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:26.624889 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:26.626696 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:26.627428 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:26.627498 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:26.627560 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:26.628365 (+ 805us) server_negotiation.cc:299] Negotiation successful
0619 08:21:26.628409 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4899,"thread_start_us":59,"threads_started":1}
I20260619 08:21:26.629478 17864 tablet_service.cc:1467] Processing CreateTablet for tablet 27e776624a0b412b9252c85e12ff5f70 (DEFAULT_TABLE table=test-table [id=2f66c68bd8dc49239f9e08a9ec5eaf4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:21:26.629886 17864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27e776624a0b412b9252c85e12ff5f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:21:26.633131 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:26.618565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36592 (local address 127.4.151.130:37131)
0619 08:21:26.623229 (+ 4664us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:26.623235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:26.623249 (+ 14us) server_negotiation.cc:407] Connection header received
0619 08:21:26.623290 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:26.623293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:26.623312 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:26.623402 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:21:26.629164 (+ 5762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:26.630211 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:26.631732 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:26.632599 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:26.632674 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:26.632739 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:26.632991 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 08:21:26.633045 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4624,"thread_start_us":47,"threads_started":1}
I20260619 08:21:26.633150 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:26.618066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:45871 (local address 127.0.0.1:58366)
0619 08:21:26.621814 (+ 3748us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:26.621821 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:26.621851 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:26.629165 (+ 7314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:26.629166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:26.629167 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:21:26.629499 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:26.629501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:26.630690 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:26.630691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:26.631704 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:26.631707 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:26.632820 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:26.632821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:26.632913 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:26.632917 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:21:26.632973 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 08:21:26.633029 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3737,"mutex_wait_us":19,"thread_start_us":73,"threads_started":1}
I20260619 08:21:26.633421 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:26.617926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:37131 (local address 127.0.0.1:36592)
0619 08:21:26.621707 (+ 3781us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:26.621716 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:26.621750 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:26.629025 (+ 7275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:26.629027 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:26.629029 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:21:26.629072 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:26.629074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:26.630328 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:26.630329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:26.631532 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:26.631536 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:26.632815 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:26.632816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:26.632887 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:26.632891 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:21:26.633263 (+ 372us) client_negotiation.cc:240] Negotiation successful
0619 08:21:26.633317 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3766,"thread_start_us":55,"threads_started":1}
I20260619 08:21:26.634143 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:26.618703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58366 (local address 127.4.151.129:45871)
0619 08:21:26.624065 (+ 5362us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:26.624070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:26.624081 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:21:26.624116 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:26.624119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:26.624134 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:26.624265 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:21:26.629590 (+ 5325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:26.630582 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:26.631806 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:26.632632 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:26.632691 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:26.632750 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:26.633977 (+ 1227us) server_negotiation.cc:299] Negotiation successful
0619 08:21:26.634053 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5323,"thread_start_us":47,"threads_started":1}
I20260619 08:21:26.634255 17745 tablet_service.cc:1467] Processing CreateTablet for tablet 27e776624a0b412b9252c85e12ff5f70 (DEFAULT_TABLE table=test-table [id=2f66c68bd8dc49239f9e08a9ec5eaf4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:21:26.634634 17745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27e776624a0b412b9252c85e12ff5f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:21:26.634722 17593 tablet_service.cc:1467] Processing CreateTablet for tablet 27e776624a0b412b9252c85e12ff5f70 (DEFAULT_TABLE table=test-table [id=2f66c68bd8dc49239f9e08a9ec5eaf4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:21:26.635027 17593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27e776624a0b412b9252c85e12ff5f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:21:26.639555 17980 tablet_bootstrap.cc:492] T 27e776624a0b412b9252c85e12ff5f70 P fbe3f9e99ffb41f2967e2471c010407f: Bootstrap starting.
I20260619 08:21:26.639721 17981 tablet_bootstrap.cc:492] T 27e776624a0b412b9252c85e12ff5f70 P 2f3c6fe7804e4c4b8c500f510401139e: Bootstrap starting.
I20260619 08:21:26.640386 17980 tablet_bootstrap.cc:654] T 27e776624a0b412b9252c85e12ff5f70 P fbe3f9e99ffb41f2967e2471c010407f: Neither blocks nor log segments found. Creating new log.
I20260619 08:21:26.640399 17981 tablet_bootstrap.cc:654] T 27e776624a0b412b9252c85e12ff5f70 P 2f3c6fe7804e4c4b8c500f510401139e: Neither blocks nor log segments found. Creating new log.
I20260619 08:21:26.640627 17979 tablet_bootstrap.cc:492] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d: Bootstrap starting.
I20260619 08:21:26.640681 17981 log.cc:824] T 27e776624a0b412b9252c85e12ff5f70 P 2f3c6fe7804e4c4b8c500f510401139e: Log is configured to *not* fsync() on all Append() calls
I20260619 08:21:26.641047 17980 log.cc:824] T 27e776624a0b412b9252c85e12ff5f70 P fbe3f9e99ffb41f2967e2471c010407f: Log is configured to *not* fsync() on all Append() calls
I20260619 08:21:26.641793 17979 tablet_bootstrap.cc:654] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d: Neither blocks nor log segments found. Creating new log.
I20260619 08:21:26.642417 17979 log.cc:824] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d: Log is configured to *not* fsync() on all Append() calls
I20260619 08:21:26.643368 17980 tablet_bootstrap.cc:492] T 27e776624a0b412b9252c85e12ff5f70 P fbe3f9e99ffb41f2967e2471c010407f: No bootstrap required, opened a new log
I20260619 08:21:26.643494 17980 ts_tablet_manager.cc:1397] T 27e776624a0b412b9252c85e12ff5f70 P fbe3f9e99ffb41f2967e2471c010407f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 08:21:26.646183 17980 raft_consensus.cc:348] T 27e776624a0b412b9252c85e12ff5f70 P fbe3f9e99ffb41f2967e2471c010407f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3c6fe7804e4c4b8c500f510401139e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33781 } } peers { permanent_uuid: "fbe3f9e99ffb41f2967e2471c010407f" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37131 } } peers { permanent_uuid: "35daca58769742dcb95828a1f3354c8d" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 45871 } }
I20260619 08:21:26.646736 17980 raft_consensus.cc:374] T 27e776624a0b412b9252c85e12ff5f70 P fbe3f9e99ffb41f2967e2471c010407f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:21:26.646840 17980 raft_consensus.cc:729] T 27e776624a0b412b9252c85e12ff5f70 P fbe3f9e99ffb41f2967e2471c010407f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe3f9e99ffb41f2967e2471c010407f, State: Initialized, Role: FOLLOWER
I20260619 08:21:26.647063 17980 consensus_queue.cc:260] T 27e776624a0b412b9252c85e12ff5f70 P fbe3f9e99ffb41f2967e2471c010407f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3c6fe7804e4c4b8c500f510401139e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33781 } } peers { permanent_uuid: "fbe3f9e99ffb41f2967e2471c010407f" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37131 } } peers { permanent_uuid: "35daca58769742dcb95828a1f3354c8d" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 45871 } }
I20260619 08:21:26.647498 17980 ts_tablet_manager.cc:1428] T 27e776624a0b412b9252c85e12ff5f70 P fbe3f9e99ffb41f2967e2471c010407f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 08:21:26.647681 17812 heartbeater.cc:503] Master 127.4.151.190:44341 was elected leader, sending a full tablet report...
I20260619 08:21:26.649699 17979 tablet_bootstrap.cc:492] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d: No bootstrap required, opened a new log
I20260619 08:21:26.649825 17979 ts_tablet_manager.cc:1397] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20260619 08:21:26.652491 17979 raft_consensus.cc:348] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3c6fe7804e4c4b8c500f510401139e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33781 } } peers { permanent_uuid: "fbe3f9e99ffb41f2967e2471c010407f" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37131 } } peers { permanent_uuid: "35daca58769742dcb95828a1f3354c8d" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 45871 } }
I20260619 08:21:26.652771 17979 raft_consensus.cc:374] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:21:26.652850 17979 raft_consensus.cc:729] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35daca58769742dcb95828a1f3354c8d, State: Initialized, Role: FOLLOWER
I20260619 08:21:26.653043 17979 consensus_queue.cc:260] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3c6fe7804e4c4b8c500f510401139e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33781 } } peers { permanent_uuid: "fbe3f9e99ffb41f2967e2471c010407f" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37131 } } peers { permanent_uuid: "35daca58769742dcb95828a1f3354c8d" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 45871 } }
I20260619 08:21:26.653437 17979 ts_tablet_manager.cc:1428] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260619 08:21:26.653612 17678 heartbeater.cc:503] Master 127.4.151.190:44341 was elected leader, sending a full tablet report...
I20260619 08:21:26.654132 17981 tablet_bootstrap.cc:492] T 27e776624a0b412b9252c85e12ff5f70 P 2f3c6fe7804e4c4b8c500f510401139e: No bootstrap required, opened a new log
I20260619 08:21:26.654206 17981 ts_tablet_manager.cc:1397] T 27e776624a0b412b9252c85e12ff5f70 P 2f3c6fe7804e4c4b8c500f510401139e: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20260619 08:21:26.656312 17981 raft_consensus.cc:348] T 27e776624a0b412b9252c85e12ff5f70 P 2f3c6fe7804e4c4b8c500f510401139e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3c6fe7804e4c4b8c500f510401139e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33781 } } peers { permanent_uuid: "fbe3f9e99ffb41f2967e2471c010407f" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37131 } } peers { permanent_uuid: "35daca58769742dcb95828a1f3354c8d" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 45871 } }
I20260619 08:21:26.656488 17981 raft_consensus.cc:374] T 27e776624a0b412b9252c85e12ff5f70 P 2f3c6fe7804e4c4b8c500f510401139e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:21:26.656536 17981 raft_consensus.cc:729] T 27e776624a0b412b9252c85e12ff5f70 P 2f3c6fe7804e4c4b8c500f510401139e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3c6fe7804e4c4b8c500f510401139e, State: Initialized, Role: FOLLOWER
I20260619 08:21:26.656672 17981 consensus_queue.cc:260] T 27e776624a0b412b9252c85e12ff5f70 P 2f3c6fe7804e4c4b8c500f510401139e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3c6fe7804e4c4b8c500f510401139e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33781 } } peers { permanent_uuid: "fbe3f9e99ffb41f2967e2471c010407f" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37131 } } peers { permanent_uuid: "35daca58769742dcb95828a1f3354c8d" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 45871 } }
I20260619 08:21:26.656947 17946 heartbeater.cc:503] Master 127.4.151.190:44341 was elected leader, sending a full tablet report...
I20260619 08:21:26.656940 17981 ts_tablet_manager.cc:1428] T 27e776624a0b412b9252c85e12ff5f70 P 2f3c6fe7804e4c4b8c500f510401139e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260619 08:21:26.804652 17947 tablet.cc:2369] T 27e776624a0b412b9252c85e12ff5f70 P 2f3c6fe7804e4c4b8c500f510401139e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:21:26.815804 17986 raft_consensus.cc:482] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:21:26.815966 17986 raft_consensus.cc:504] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3c6fe7804e4c4b8c500f510401139e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33781 } } peers { permanent_uuid: "fbe3f9e99ffb41f2967e2471c010407f" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37131 } } peers { permanent_uuid: "35daca58769742dcb95828a1f3354c8d" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 45871 } }
I20260619 08:21:26.816459 17986 leader_election.cc:288] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f3c6fe7804e4c4b8c500f510401139e (127.4.151.131:33781), fbe3f9e99ffb41f2967e2471c010407f (127.4.151.130:37131)
I20260619 08:21:26.820987 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:26.816873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:47969 (local address 127.4.151.130:37131)
0619 08:21:26.816917 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:26.816920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:26.817085 (+ 165us) server_negotiation.cc:407] Connection header received
0619 08:21:26.817204 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:26.817205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:26.817215 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:26.817248 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:21:26.817560 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:26.818316 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:26.819926 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:26.820538 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:26.820581 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:26.820590 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:26.820887 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 08:21:26.820927 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:21:26.820987 17976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:26.816870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:44815 (local address 127.4.151.131:33781)
0619 08:21:26.816917 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:26.816920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:26.817089 (+ 169us) server_negotiation.cc:407] Connection header received
0619 08:21:26.817203 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:26.817204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:26.817215 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:26.817258 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:21:26.817560 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:26.818324 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:26.819926 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:26.820524 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:26.820581 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:26.820590 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:26.820867 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 08:21:26.820913 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:21:26.821254 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:26.816632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:37131 (local address 127.4.151.129:47969)
0619 08:21:26.817008 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:26.817022 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:26.817119 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:26.817324 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:26.817325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:26.817327 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:21:26.817473 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:26.817475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:26.818425 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:26.818426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:26.819807 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:26.819811 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:26.820714 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:26.820714 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:26.820785 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:26.820789 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:21:26.821081 (+ 292us) client_negotiation.cc:240] Negotiation successful
0619 08:21:26.821142 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":57,"thread_start_us":110,"threads_started":1}
I20260619 08:21:26.821652 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:26.816633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:33781 (local address 127.4.151.129:44815)
0619 08:21:26.817034 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:26.817045 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:26.817119 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:26.817324 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:26.817325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:26.817328 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:21:26.817472 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:26.817476 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:26.818425 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:26.818426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:26.819807 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:26.819811 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:26.820699 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:26.820700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:26.820774 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:26.820779 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:21:26.821508 (+ 729us) client_negotiation.cc:240] Negotiation successful
0619 08:21:26.821555 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":126,"spinlock_wait_cycles":10624,"thread_start_us":121,"threads_started":1}
I20260619 08:21:26.821708 17766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e776624a0b412b9252c85e12ff5f70" candidate_uuid: "35daca58769742dcb95828a1f3354c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe3f9e99ffb41f2967e2471c010407f" is_pre_election: true
I20260619 08:21:26.821892 17766 raft_consensus.cc:2454] T 27e776624a0b412b9252c85e12ff5f70 P fbe3f9e99ffb41f2967e2471c010407f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35daca58769742dcb95828a1f3354c8d in term 0.
I20260619 08:21:26.822108 17565 leader_election.cc:302] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35daca58769742dcb95828a1f3354c8d, fbe3f9e99ffb41f2967e2471c010407f; no voters:
I20260619 08:21:26.822124 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e776624a0b412b9252c85e12ff5f70" candidate_uuid: "35daca58769742dcb95828a1f3354c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3c6fe7804e4c4b8c500f510401139e" is_pre_election: true
I20260619 08:21:26.822232 17986 raft_consensus.cc:2781] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:21:26.822281 17900 raft_consensus.cc:2454] T 27e776624a0b412b9252c85e12ff5f70 P 2f3c6fe7804e4c4b8c500f510401139e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35daca58769742dcb95828a1f3354c8d in term 0.
I20260619 08:21:26.822295 17986 raft_consensus.cc:482] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:21:26.822320 17986 raft_consensus.cc:3037] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:21:26.823117 17986 raft_consensus.cc:504] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3c6fe7804e4c4b8c500f510401139e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33781 } } peers { permanent_uuid: "fbe3f9e99ffb41f2967e2471c010407f" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37131 } } peers { permanent_uuid: "35daca58769742dcb95828a1f3354c8d" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 45871 } }
I20260619 08:21:26.823320 17986 leader_election.cc:288] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d [CANDIDATE]: Term 1 election: Requested vote from peers 2f3c6fe7804e4c4b8c500f510401139e (127.4.151.131:33781), fbe3f9e99ffb41f2967e2471c010407f (127.4.151.130:37131)
I20260619 08:21:26.823465 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e776624a0b412b9252c85e12ff5f70" candidate_uuid: "35daca58769742dcb95828a1f3354c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3c6fe7804e4c4b8c500f510401139e"
I20260619 08:21:26.823555 17900 raft_consensus.cc:3037] T 27e776624a0b412b9252c85e12ff5f70 P 2f3c6fe7804e4c4b8c500f510401139e [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:21:26.823588 17766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e776624a0b412b9252c85e12ff5f70" candidate_uuid: "35daca58769742dcb95828a1f3354c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe3f9e99ffb41f2967e2471c010407f"
I20260619 08:21:26.823663 17766 raft_consensus.cc:3037] T 27e776624a0b412b9252c85e12ff5f70 P fbe3f9e99ffb41f2967e2471c010407f [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:21:26.824385 17900 raft_consensus.cc:2454] T 27e776624a0b412b9252c85e12ff5f70 P 2f3c6fe7804e4c4b8c500f510401139e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35daca58769742dcb95828a1f3354c8d in term 1.
I20260619 08:21:26.824465 17766 raft_consensus.cc:2454] T 27e776624a0b412b9252c85e12ff5f70 P fbe3f9e99ffb41f2967e2471c010407f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35daca58769742dcb95828a1f3354c8d in term 1.
I20260619 08:21:26.824580 17566 leader_election.cc:302] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3c6fe7804e4c4b8c500f510401139e, 35daca58769742dcb95828a1f3354c8d; no voters:
I20260619 08:21:26.824669 17986 raft_consensus.cc:2781] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:21:26.824817 17986 raft_consensus.cc:686] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d [term 1 LEADER]: Becoming Leader. State: Replica: 35daca58769742dcb95828a1f3354c8d, State: Running, Role: LEADER
I20260619 08:21:26.824913 17986 consensus_queue.cc:237] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3c6fe7804e4c4b8c500f510401139e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33781 } } peers { permanent_uuid: "fbe3f9e99ffb41f2967e2471c010407f" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37131 } } peers { permanent_uuid: "35daca58769742dcb95828a1f3354c8d" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 45871 } }
I20260619 08:21:26.825817 17479 catalog_manager.cc:5495] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 35daca58769742dcb95828a1f3354c8d (127.4.151.129). New cstate: current_term: 1 leader_uuid: "35daca58769742dcb95828a1f3354c8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3c6fe7804e4c4b8c500f510401139e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbe3f9e99ffb41f2967e2471c010407f" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35daca58769742dcb95828a1f3354c8d" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 45871 } health_report { overall_health: HEALTHY } } }
I20260619 08:21:26.850073 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:26.846814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58372 (local address 127.4.151.129:45871)
0619 08:21:26.846873 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:26.846875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:26.846883 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:21:26.846928 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:26.846929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:26.846947 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:26.847025 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:21:26.847180 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:26.847876 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:26.848971 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:26.849289 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:26.849367 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:26.849626 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:21:26.849796 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:21:26.849892 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:26.849913 (+ 21us) server_negotiation.cc:299] Negotiation successful
0619 08:21:26.849938 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 08:21:26.852293 17900 raft_consensus.cc:1261] T 27e776624a0b412b9252c85e12ff5f70 P 2f3c6fe7804e4c4b8c500f510401139e [term 1 FOLLOWER]: Refusing update from remote peer 35daca58769742dcb95828a1f3354c8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:21:26.852275 17766 raft_consensus.cc:1261] T 27e776624a0b412b9252c85e12ff5f70 P fbe3f9e99ffb41f2967e2471c010407f [term 1 FOLLOWER]: Refusing update from remote peer 35daca58769742dcb95828a1f3354c8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:21:26.852723 17990 consensus_queue.cc:1035] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe3f9e99ffb41f2967e2471c010407f" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:21:26.852962 17986 consensus_queue.cc:1035] T 27e776624a0b412b9252c85e12ff5f70 P 35daca58769742dcb95828a1f3354c8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3c6fe7804e4c4b8c500f510401139e" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:21:26.857779 17993 mvcc.cc:205] Tried to move back new op lower bound from 7298487446944010240 to 7298487446835576832. Current Snapshot: MvccSnapshot[applied={T|T < 7298487446944010240}]
I20260619 08:21:26.859018 17994 mvcc.cc:205] Tried to move back new op lower bound from 7298487446944010240 to 7298487446835576832. Current Snapshot: MvccSnapshot[applied={T|T < 7298487446944010240}]
I20260619 08:21:26.860509 17480 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:21:26.860642 17480 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:21:26.861581 17480 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d14fa17612c4563aa57cf05be686573: Sending DeleteTablet for 3 replicas of tablet 27e776624a0b412b9252c85e12ff5f70
I20260619 08:21:26.861994 17593 tablet_service.cc:1514] Processing DeleteTablet for tablet 27e776624a0b412b9252c85e12ff5f70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:21:26 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:58366
I20260619 08:21:26.862012 17745 tablet_service.cc:1514] Processing DeleteTablet for tablet 27e776624a0b412b9252c85e12ff5f70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:21:26 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:36592
I20260619 08:21:26.861994 17864 tablet_service.cc:1514] Processing DeleteTablet for tablet 27e776624a0b412b9252c85e12ff5f70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:21:26 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:40492
I20260619 08:21:26.862581 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 17518
I20260619 08:21:26.869503 18004 tablet_replica.cc:330] T 27e776624a0b412b9252c85e12ff5f70 P fbe3f9e99ffb41f2967e2471c010407f: stopping tablet replica
I20260619 08:21:26.869688 18004 raft_consensus.cc:2229] T 27e776624a0b412b9252c85e12ff5f70 P fbe3f9e99ffb41f2967e2471c010407f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:21:26.869822 18004 raft_consensus.cc:2258] T 27e776624a0b412b9252c85e12ff5f70 P fbe3f9e99ffb41f2967e2471c010407f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:21:26.870005 18003 tablet_replica.cc:330] T 27e776624a0b412b9252c85e12ff5f70 P 2f3c6fe7804e4c4b8c500f510401139e: stopping tablet replica
I20260619 08:21:26.870117 18003 raft_consensus.cc:2229] T 27e776624a0b412b9252c85e12ff5f70 P 2f3c6fe7804e4c4b8c500f510401139e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:21:26.870219 18003 raft_consensus.cc:2258] T 27e776624a0b412b9252c85e12ff5f70 P 2f3c6fe7804e4c4b8c500f510401139e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:21:26.870435 18004 ts_tablet_manager.cc:1905] T 27e776624a0b412b9252c85e12ff5f70 P fbe3f9e99ffb41f2967e2471c010407f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:21:26.872303 18004 ts_tablet_manager.cc:1918] T 27e776624a0b412b9252c85e12ff5f70 P fbe3f9e99ffb41f2967e2471c010407f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:21:26.872453 18004 log.cc:1196] T 27e776624a0b412b9252c85e12ff5f70 P fbe3f9e99ffb41f2967e2471c010407f: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/27e776624a0b412b9252c85e12ff5f70
I20260619 08:21:26.873039 18004 ts_tablet_manager.cc:1939] T 27e776624a0b412b9252c85e12ff5f70 P fbe3f9e99ffb41f2967e2471c010407f: Deleting consensus metadata
I20260619 08:21:26.873597 17464 catalog_manager.cc:4841] TS fbe3f9e99ffb41f2967e2471c010407f (127.4.151.130:37131): tablet 27e776624a0b412b9252c85e12ff5f70 (table test-table [id=2f66c68bd8dc49239f9e08a9ec5eaf4d]) successfully deleted
I20260619 08:21:26.874194 18003 ts_tablet_manager.cc:1905] T 27e776624a0b412b9252c85e12ff5f70 P 2f3c6fe7804e4c4b8c500f510401139e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:21:26.876020 18003 ts_tablet_manager.cc:1918] T 27e776624a0b412b9252c85e12ff5f70 P 2f3c6fe7804e4c4b8c500f510401139e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:21:26.876101 18003 log.cc:1196] T 27e776624a0b412b9252c85e12ff5f70 P 2f3c6fe7804e4c4b8c500f510401139e: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/27e776624a0b412b9252c85e12ff5f70
I20260619 08:21:26.876379 18003 ts_tablet_manager.cc:1939] T 27e776624a0b412b9252c85e12ff5f70 P 2f3c6fe7804e4c4b8c500f510401139e: Deleting consensus metadata
I20260619 08:21:26.877033 17465 catalog_manager.cc:4841] TS 2f3c6fe7804e4c4b8c500f510401139e (127.4.151.131:33781): tablet 27e776624a0b412b9252c85e12ff5f70 (table test-table [id=2f66c68bd8dc49239f9e08a9ec5eaf4d]) successfully deleted
W20260619 08:21:26.877457 17466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.151.129:45871 (error 108)
I20260619 08:21:26.877707 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 17682
I20260619 08:21:26.884584 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 17815
I20260619 08:21:26.885241 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:26.884723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:45871 (local address 127.0.0.1:58382)
0619 08:21:26.885151 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:26.885199 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.129:45871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406}
W20260619 08:21:26.886509 17466 catalog_manager.cc:4584] TS 35daca58769742dcb95828a1f3354c8d (127.4.151.129:45871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 27e776624a0b412b9252c85e12ff5f70: Network error: Client connection negotiation failed: client connection to 127.4.151.129:45871: connect: Connection refused (error 111)
I20260619 08:21:26.890377 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 17447
I20260619 08:21:26.901238 4702 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 08:21:27.246819 4702 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 08:21:27.246924 4702 mini_ranger.cc:66] Stopping Ranger...
I20260619 08:21:27.592533 4702 mini_ranger.cc:68] Stopped Ranger
2026-06-19 08:21:27.592 UTC [16514] LOG: received smart shutdown request
2026-06-19 08:21:27.597 UTC [16514] LOG: background worker "logical replication launcher" (PID 16523) exited with exit code 1
2026-06-19 08:21:27.597 UTC [16518] LOG: shutting down
2026-06-19 08:21:27.858 UTC [16514] LOG: database system is shut down
2026-06-19T08:21:27Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51370 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:21:28 dist-test-slave-d3n8 krb5kdc[18016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:21:28 dist-test-slave-d3n8 krb5kdc[18016](info): set up 2 sockets
Jun 19 08:21:28 dist-test-slave-d3n8 krb5kdc[18016](info): commencing operation
krb5kdc: starting...
W20260619 08:21:29.053619 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:21:29 dist-test-slave-d3n8 krb5kdc[18016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:21:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:21:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:21:29.189913 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:33791
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:33791,127.4.151.189:34817,127.4.151.188:44523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:21:29.277529 18029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:29.277761 18029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:29.277787 18029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:29.279291 18029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:21:29.279366 18029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:29.279382 18029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:29.279395 18029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:21:29.279408 18029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:21:29.281276 18029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:33791,127.4.151.189:34817,127.4.151.188:44523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:33791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:29.281651 18029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:29.281976 18029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:29.288230 18034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:29.288429 18029 server_base.cc:1034] running on GCE node
W20260619 08:21:29.288214 18035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:29.288214 18037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:29.288969 18029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:29.289449 18029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:29.290635 18029 hybrid_clock.cc:648] HybridClock initialized: now 1781857289290618 us; error 34 us; skew 500 ppm
Jun 19 08:21:29 dist-test-slave-d3n8 krb5kdc[18016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857289, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:29.293674 18029 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:21:29.294179 18029 webserver.cc:460] Webserver started at http://127.4.151.190:40491/ using document root <none> and password file <none>
I20260619 08:21:29.294437 18029 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:29.294484 18029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:21:29.294641 18029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:21:29.295894 18029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "84c476ed181c41888d52f69d62a50784"
format_stamp: "Formatted at 2026-06-19 08:21:29 on dist-test-slave-d3n8"
server_key: "95c8460ecde39826c22e2d7b0d24038c"
server_key_iv: "bada2d0b11b6458e50a394cc78631fe6"
server_key_version: "encryptionkey@0"
I20260619 08:21:29.296401 18029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "84c476ed181c41888d52f69d62a50784"
format_stamp: "Formatted at 2026-06-19 08:21:29 on dist-test-slave-d3n8"
server_key: "95c8460ecde39826c22e2d7b0d24038c"
server_key_iv: "bada2d0b11b6458e50a394cc78631fe6"
server_key_version: "encryptionkey@0"
I20260619 08:21:29.298197 18029 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 08:21:29.299202 18044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:29.299440 18029 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:21:29.299541 18029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "84c476ed181c41888d52f69d62a50784"
format_stamp: "Formatted at 2026-06-19 08:21:29 on dist-test-slave-d3n8"
server_key: "95c8460ecde39826c22e2d7b0d24038c"
server_key_iv: "bada2d0b11b6458e50a394cc78631fe6"
server_key_version: "encryptionkey@0"
I20260619 08:21:29.299619 18029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:29.320197 18029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:29.322652 18029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:29.322865 18029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:29.327307 18029 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:33791
I20260619 08:21:29.327354 18096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:33791 every 8 connection(s)
I20260619 08:21:29.327723 18029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:21:29.328567 18097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:21:29.330384 18097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33791 } has no permanent_uuid. Determining permanent_uuid...
I20260619 08:21:29.336167 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 18029
I20260619 08:21:29.336506 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:21:29.336802 4702 external_mini_cluster.cc:1428] Setting key bfe26c24e7c9b20ce8040751270e29a6
Jun 19 08:21:29 dist-test-slave-d3n8 krb5kdc[18016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857289, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:21:29.344779 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.331115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33791 (local address 127.0.0.1:34072)
0619 08:21:29.331513 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.331558 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:29.332072 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:29.332584 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:29.332589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:29.333002 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:29.333243 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:29.333250 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.334711 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.334713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:29.335718 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:29.335722 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.336959 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.336960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:29.337069 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:29.337840 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:29.337853 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:29.340083 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:29.342823 (+ 2740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:29.342840 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:29.342843 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:29.343065 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:29.343427 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:29.343428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:29.343428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:29.343515 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:29.344047 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:29.344054 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:29.344119 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 08:21:29.344299 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 08:21:29.344599 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":144,"thread_start_us":84,"threads_started":1}
I20260619 08:21:29.344786 18100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.331312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34072 (local address 127.4.151.190:33791)
0619 08:21:29.331677 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:29.331685 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:29.331744 (+ 59us) server_negotiation.cc:407] Connection header received
0619 08:21:29.332328 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:29.332331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:29.332351 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:29.332489 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:29.333678 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.334561 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.335861 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.336774 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.336937 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:29.340246 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:29.340256 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:29.340256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:29.340267 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:29.342706 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.343229 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.343232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.343233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.343351 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.343589 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.343590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.343591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.343919 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:29.344024 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:29.344352 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 08:21:29.344523 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":314,"thread_start_us":258,"threads_started":1}
I20260619 08:21:29.345693 18097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 } has no permanent_uuid. Determining permanent_uuid...
I20260619 08:21:29.346076 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.345956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:34817 (local address 127.0.0.1:38176)
0619 08:21:29.346006 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.346054 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.189:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
WARNING: no policy specified for kudu/127.4.151.189@KRBTEST.COM; defaulting to no policy
W20260619 08:21:29.346252 18048 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.151.189:34817: connect: Connection refused (error 111)
I20260619 08:21:29.346825 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.346693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:34817 (local address 127.0.0.1:38182)
0619 08:21:29.346768 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.346802 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.189:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 08:21:29.346995 18097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.151.189:34817: Network error: Client connection negotiation failed: client connection to 127.4.151.189:34817: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.189.keytab.
Entry for principal kudu/127.4.151.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.189.keytab.
I20260619 08:21:29.369297 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.189:34817
--webserver_interface=127.4.151.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.189.keytab
--principal=kudu/127.4.151.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:33791,127.4.151.189:34817,127.4.151.188:44523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:21:29.396411 18097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 } attempt: 1
I20260619 08:21:29.396975 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.396817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:34817 (local address 127.0.0.1:38190)
0619 08:21:29.396907 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.396947 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.189:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 08:21:29.397358 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.397227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:34817 (local address 127.0.0.1:38206)
0619 08:21:29.397315 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.397339 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.189:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260619 08:21:29.397478 18097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.151.189:34817: Network error: Client connection negotiation failed: client connection to 127.4.151.189:34817: connect: Connection refused (error 111)
W20260619 08:21:29.457213 18104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:29.457470 18104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:29.457495 18104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:29.458755 18104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:21:29.458806 18104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:29.458822 18104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:29.458834 18104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:21:29.458846 18104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:21:29.460494 18104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:33791,127.4.151.189:34817,127.4.151.188:44523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.189.keytab
--principal=kudu/127.4.151.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.189:34817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.151.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:29.460803 18104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:29.461052 18104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:21:29.465793 18097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 } attempt: 2
I20260619 08:21:29.466395 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.466200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:34817 (local address 127.0.0.1:38220)
0619 08:21:29.466279 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.466313 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.189:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 08:21:29.466634 18110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:29.466631 18109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:29.466710 18104 server_base.cc:1034] running on GCE node
I20260619 08:21:29.466804 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.466656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:34817 (local address 127.0.0.1:38222)
0619 08:21:29.466751 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.466776 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.189:34817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260619 08:21:29.466904 18097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.151.189:34817: Network error: Client connection negotiation failed: client connection to 127.4.151.189:34817: connect: Connection refused (error 111)
W20260619 08:21:29.466894 18112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:29.467161 18104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:29.467577 18104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:29.468773 18104 hybrid_clock.cc:648] HybridClock initialized: now 1781857289468753 us; error 38 us; skew 500 ppm
Jun 19 08:21:29 dist-test-slave-d3n8 krb5kdc[18016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857289, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:29.472052 18104 init.cc:377] Logged in from keytab as kudu/127.4.151.189@KRBTEST.COM (short username kudu)
I20260619 08:21:29.472708 18104 webserver.cc:460] Webserver started at http://127.4.151.189:46321/ using document root <none> and password file <none>
I20260619 08:21:29.472985 18104 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:29.473058 18104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:21:29.473219 18104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:21:29.474421 18104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-1/data/instance:
uuid: "b6647a2cc6eb45e8bff2fad5f3fa68b7"
format_stamp: "Formatted at 2026-06-19 08:21:29 on dist-test-slave-d3n8"
server_key: "f306b38b02475250a903cf7849f63466"
server_key_iv: "0c3dc133996b630630f5152194484b29"
server_key_version: "encryptionkey@0"
I20260619 08:21:29.474824 18104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-1/wal/instance:
uuid: "b6647a2cc6eb45e8bff2fad5f3fa68b7"
format_stamp: "Formatted at 2026-06-19 08:21:29 on dist-test-slave-d3n8"
server_key: "f306b38b02475250a903cf7849f63466"
server_key_iv: "0c3dc133996b630630f5152194484b29"
server_key_version: "encryptionkey@0"
I20260619 08:21:29.476476 18104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:21:29.477510 18119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:29.478992 18104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 08:21:29.479080 18104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-1/wal
uuid: "b6647a2cc6eb45e8bff2fad5f3fa68b7"
format_stamp: "Formatted at 2026-06-19 08:21:29 on dist-test-slave-d3n8"
server_key: "f306b38b02475250a903cf7849f63466"
server_key_iv: "0c3dc133996b630630f5152194484b29"
server_key_version: "encryptionkey@0"
I20260619 08:21:29.479179 18104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:29.490511 18104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:29.493012 18104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:29.493180 18104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:29.497771 18104 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.189:34817
I20260619 08:21:29.497850 18171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.189:34817 every 8 connection(s)
I20260619 08:21:29.498353 18104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-1/data/info.pb
I20260619 08:21:29.499238 18172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:21:29.500931 18172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33791 } has no permanent_uuid. Determining permanent_uuid...
I20260619 08:21:29.505105 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 18104
I20260619 08:21:29.505250 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-1/wal/instance
I20260619 08:21:29.505489 4702 external_mini_cluster.cc:1428] Setting key d92c99a1286d787a8329e55263dc1e4c
Jun 19 08:21:29 dist-test-slave-d3n8 krb5kdc[18016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857289, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.189@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:21:29.511366 18100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.501793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34078 (local address 127.4.151.190:33791)
0619 08:21:29.501876 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:29.501879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:29.502144 (+ 265us) server_negotiation.cc:407] Connection header received
0619 08:21:29.502624 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:29.502626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:29.502642 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:29.502694 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:29.503394 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.503923 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.504965 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.505185 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.505279 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:29.507807 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:29.507808 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:29.507809 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:29.507818 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:29.509705 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.510200 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.510202 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.510202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.510273 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.510578 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.510580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.510580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.510760 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:29.510842 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:29.511117 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 08:21:29.511221 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 08:21:29.511394 18174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.501765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33791 (local address 127.0.0.1:34078)
0619 08:21:29.501941 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.501979 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:29.502495 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:29.502776 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:29.502779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:29.503074 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:29.503280 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:29.503285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.504059 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.504060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:29.504864 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:29.504866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.505292 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.505293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:29.505491 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:29.506156 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:29.506165 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:29.507690 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:29.509806 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:29.509814 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:29.509816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:29.510087 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:29.510369 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:29.510370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:29.510371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:29.510493 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:29.510841 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:29.510847 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:29.510976 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 08:21:29.511071 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 08:21:29.511222 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260619 08:21:29.512073 18172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.151.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.188@KRBTEST.COM" created.
Jun 19 08:21:29 dist-test-slave-d3n8 krb5kdc[18016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857289, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.189@KRBTEST.COM for kudu/127.4.151.189@KRBTEST.COM
I20260619 08:21:29.521356 18174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.512371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:34817 (local address 127.0.0.1:38224)
0619 08:21:29.512497 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.512515 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:29.512611 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:29.513063 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:29.513064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:29.513293 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:29.513346 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:29.513349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.514379 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.514380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:29.514884 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:29.514885 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.515400 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.515401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:29.515471 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:29.515939 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:29.515944 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:29.517891 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:29.520162 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:29.520163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:29.520164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:29.520316 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:29.520569 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:29.520569 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:29.520570 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:29.520624 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:29.520970 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:29.520971 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:29.521010 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 08:21:29.521099 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 08:21:29.521211 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 08:21:29.521407 18176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.512625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38224 (local address 127.4.151.189:34817)
0619 08:21:29.512863 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:29.512867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:29.512888 (+ 21us) server_negotiation.cc:407] Connection header received
0619 08:21:29.512910 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:29.512912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:29.512925 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:29.513045 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:29.513475 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.514278 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.514984 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.515287 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.515403 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:29.517999 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:29.518008 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:29.518008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:29.518016 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:29.520010 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.520419 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.520422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.520423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.520482 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.520689 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.520690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.520690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.520906 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:29.520990 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:29.521147 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 08:21:29.521281 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":204,"thread_start_us":103,"threads_started":1}
I20260619 08:21:29.522109 18172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } has no permanent_uuid. Determining permanent_uuid...
I20260619 08:21:29.522810 18174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.522636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:44523 (local address 127.0.0.1:44264)
0619 08:21:29.522720 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.522779 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 08:21:29.522979 18122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.151.188:44523: connect: Connection refused (error 111)
I20260619 08:21:29.523675 18174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.523446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:44523 (local address 127.0.0.1:44278)
0619 08:21:29.523604 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.523645 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260619 08:21:29.523797 18172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.151.188:44523: Network error: Client connection negotiation failed: client connection to 127.4.151.188:44523: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.188.keytab.
Entry for principal kudu/127.4.151.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.188.keytab.
I20260619 08:21:29.537962 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.188:44523
--webserver_interface=127.4.151.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.188.keytab
--principal=kudu/127.4.151.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:33791,127.4.151.189:34817,127.4.151.188:44523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:21:29.558064 18097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 } attempt: 3
Jun 19 08:21:29 dist-test-slave-d3n8 krb5kdc[18016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857289, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for kudu/127.4.151.189@KRBTEST.COM
I20260619 08:21:29.565536 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.558437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:34817 (local address 127.0.0.1:38232)
0619 08:21:29.558505 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.558517 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:29.558570 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:29.558836 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:29.558838 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:29.559008 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:29.559075 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:29.559076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.559789 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.559790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:29.560325 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:29.560327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.560807 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.560808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:29.560885 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:29.561446 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:29.561455 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:29.562808 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:29.564400 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:29.564401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:29.564402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:29.564580 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:29.564831 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:29.564831 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:29.564832 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:29.564891 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:29.565156 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:29.565157 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:29.565216 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 08:21:29.565302 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 08:21:29.565429 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 08:21:29.565532 18176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.558645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38232 (local address 127.4.151.189:34817)
0619 08:21:29.558711 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:29.558713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:29.558721 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:21:29.558748 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:29.558749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:29.558759 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:29.558829 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:29.559155 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.559700 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.560436 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.560725 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.560812 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:29.562903 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:29.562905 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:29.562906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:29.562913 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:29.564290 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.564705 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.564707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.564707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.564759 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.564964 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.564965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.564965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.565086 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:29.565166 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:29.565327 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 08:21:29.565434 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 08:21:29.565981 18097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } has no permanent_uuid. Determining permanent_uuid...
I20260619 08:21:29.566365 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.566264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:44523 (local address 127.0.0.1:44294)
0619 08:21:29.566315 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.566345 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 08:21:29.566674 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.566580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:44523 (local address 127.0.0.1:44308)
0619 08:21:29.566630 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.566653 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 08:21:29.566772 18097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.151.188:44523: Network error: Client connection negotiation failed: client connection to 127.4.151.188:44523: connect: Connection refused (error 111)
I20260619 08:21:29.573041 18172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } attempt: 1
I20260619 08:21:29.573572 18174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.573433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:44523 (local address 127.0.0.1:44312)
0619 08:21:29.573475 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.573527 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260619 08:21:29.573935 18174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.573846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:44523 (local address 127.0.0.1:44322)
0619 08:21:29.573889 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.573913 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 08:21:29.574040 18172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.151.188:44523: Network error: Client connection negotiation failed: client connection to 127.4.151.188:44523: connect: Connection refused (error 111)
I20260619 08:21:29.598002 18097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } attempt: 1
I20260619 08:21:29.598505 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.598354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:44523 (local address 127.0.0.1:44328)
0619 08:21:29.598434 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.598476 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 08:21:29.598886 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.598790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:44523 (local address 127.0.0.1:44342)
0619 08:21:29.598837 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.598862 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 08:21:29.598978 18097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.151.188:44523: Network error: Client connection negotiation failed: client connection to 127.4.151.188:44523: connect: Connection refused (error 111)
W20260619 08:21:29.629098 18179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:29.629312 18179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:29.629336 18179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:29.630673 18179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:21:29.630724 18179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:29.630739 18179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:29.630751 18179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:21:29.630764 18179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:21:29.632339 18179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:33791,127.4.151.189:34817,127.4.151.188:44523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.188.keytab
--principal=kudu/127.4.151.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.188:44523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.151.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:29.632608 18179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:29.632881 18179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:29.638217 18185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:29.638293 18184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:29.638167 18187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:29.638439 18179 server_base.cc:1034] running on GCE node
I20260619 08:21:29.638711 18179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:29.638991 18179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:29.640247 18179 hybrid_clock.cc:648] HybridClock initialized: now 1781857289640223 us; error 46 us; skew 500 ppm
I20260619 08:21:29.642244 18172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } attempt: 2
I20260619 08:21:29.642673 18174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.642520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:44523 (local address 127.0.0.1:44352)
0619 08:21:29.642584 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.642622 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
Jun 19 08:21:29 dist-test-slave-d3n8 krb5kdc[18016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857289, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:29.643069 18174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.642941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:44523 (local address 127.0.0.1:44356)
0619 08:21:29.643004 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.643039 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 08:21:29.643190 18172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.151.188:44523: Network error: Client connection negotiation failed: client connection to 127.4.151.188:44523: connect: Connection refused (error 111)
I20260619 08:21:29.643177 18179 init.cc:377] Logged in from keytab as kudu/127.4.151.188@KRBTEST.COM (short username kudu)
I20260619 08:21:29.643667 18179 webserver.cc:460] Webserver started at http://127.4.151.188:44033/ using document root <none> and password file <none>
I20260619 08:21:29.643914 18179 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:29.643961 18179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:21:29.644091 18179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:21:29.645217 18179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-2/data/instance:
uuid: "ac03e24ac2c04c0fb57a44795f3b9032"
format_stamp: "Formatted at 2026-06-19 08:21:29 on dist-test-slave-d3n8"
server_key: "01aee1b45826a973f676cf33965d2849"
server_key_iv: "42479196e16aa19a922a2b918a0727f2"
server_key_version: "encryptionkey@0"
I20260619 08:21:29.645645 18179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-2/wal/instance:
uuid: "ac03e24ac2c04c0fb57a44795f3b9032"
format_stamp: "Formatted at 2026-06-19 08:21:29 on dist-test-slave-d3n8"
server_key: "01aee1b45826a973f676cf33965d2849"
server_key_iv: "42479196e16aa19a922a2b918a0727f2"
server_key_version: "encryptionkey@0"
I20260619 08:21:29.647063 18179 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:21:29.648172 18194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:29.648527 18179 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:21:29.648676 18179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-2/wal
uuid: "ac03e24ac2c04c0fb57a44795f3b9032"
format_stamp: "Formatted at 2026-06-19 08:21:29 on dist-test-slave-d3n8"
server_key: "01aee1b45826a973f676cf33965d2849"
server_key_iv: "42479196e16aa19a922a2b918a0727f2"
server_key_version: "encryptionkey@0"
I20260619 08:21:29.648759 18179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:29.674218 18097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } attempt: 2
I20260619 08:21:29.674710 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.674546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:44523 (local address 127.0.0.1:44364)
0619 08:21:29.674620 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.674680 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 08:21:29.675019 18179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:29.675081 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.674981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:44523 (local address 127.0.0.1:44366)
0619 08:21:29.675030 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.675054 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 08:21:29.675168 18097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.151.188:44523: Network error: Client connection negotiation failed: client connection to 127.4.151.188:44523: connect: Connection refused (error 111)
I20260619 08:21:29.678937 18179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:29.679082 18179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:29.684723 18179 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.188:44523
I20260619 08:21:29.684952 18246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.188:44523 every 8 connection(s)
I20260619 08:21:29.685178 18179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-2/data/info.pb
I20260619 08:21:29.686687 18247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:21:29.688575 18247 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33791 } has no permanent_uuid. Determining permanent_uuid...
I20260619 08:21:29.694109 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 18179
I20260619 08:21:29.694227 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781857130600961-4702-0/minicluster-data/master-2/wal/instance
I20260619 08:21:29.694471 4702 external_mini_cluster.cc:1428] Setting key 2b84cb9e720c8359dc5ce519bc770263
Jun 19 08:21:29 dist-test-slave-d3n8 krb5kdc[18016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857289, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.188@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:21:29.700738 18249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.689455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33791 (local address 127.0.0.1:34092)
0619 08:21:29.689677 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.689725 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:29.690237 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:29.690512 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:29.690515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:29.690807 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:29.691022 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:29.691027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.691742 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.691743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:29.692724 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:29.692727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.693538 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.693539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:29.693686 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:29.694490 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:29.694503 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:29.696546 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:29.698921 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:29.698928 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:29.698930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:29.699277 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:29.699578 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:29.699579 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:29.699579 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:29.699740 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:29.700104 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:29.700111 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:29.700307 (+ 196us) client_negotiation.cc:769] Sending connection context
0619 08:21:29.700424 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 08:21:29.700558 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":134,"threads_started":1}
I20260619 08:21:29.700742 18100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.689513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34092 (local address 127.4.151.190:33791)
0619 08:21:29.689632 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:29.689637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:29.689839 (+ 202us) server_negotiation.cc:407] Connection header received
0619 08:21:29.690369 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:29.690370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:29.690381 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:29.690454 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:29.691129 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.691632 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.692969 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.693263 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.693529 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:29.696651 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:29.696654 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:29.696655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:29.696667 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:29.698810 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.699413 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.699415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.699415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.699489 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.699837 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.699838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.699838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.699967 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:29.700108 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:29.700495 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 08:21:29.700611 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
Jun 19 08:21:29 dist-test-slave-d3n8 krb5kdc[18016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.188@KRBTEST.COM
I20260619 08:21:29.701879 18247 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 } has no permanent_uuid. Determining permanent_uuid...
I20260619 08:21:29.705476 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.694972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44382 (local address 127.4.151.188:44523)
0619 08:21:29.695171 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:29.695177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:29.695270 (+ 93us) server_negotiation.cc:407] Connection header received
0619 08:21:29.695348 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:29.695351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:29.695371 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:29.695447 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:29.696246 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.697294 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.698240 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.698569 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.698686 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:29.701697 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:29.701704 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:29.701704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:29.701712 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:29.703911 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.704279 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.704283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.704283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.704354 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.704601 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.704602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.704602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.704790 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:29.704986 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:29.705151 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 08:21:29.705337 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":105,"threads_started":1}
Jun 19 08:21:29 dist-test-slave-d3n8 krb5kdc[18016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857289, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.188@KRBTEST.COM for kudu/127.4.151.189@KRBTEST.COM
I20260619 08:21:29.710834 18249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.702330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:34817 (local address 127.0.0.1:38246)
0619 08:21:29.702708 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.702719 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:29.702801 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:29.702953 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:29.702953 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:29.703168 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:29.703216 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:29.703218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.704050 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.704051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:29.704815 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:29.704817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.705351 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.705352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:29.705454 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:29.705907 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:29.705912 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:29.707220 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:29.709244 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:29.709246 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:29.709246 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:29.709515 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:29.709931 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:29.709931 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:29.709932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:29.710149 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:29.710568 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:29.710569 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:29.710611 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 08:21:29.710677 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 08:21:29.710761 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363}
I20260619 08:21:29.710973 18176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.702397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38246 (local address 127.4.151.189:34817)
0619 08:21:29.702482 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:29.702485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:29.702815 (+ 330us) server_negotiation.cc:407] Connection header received
0619 08:21:29.702873 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:29.702874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:29.702889 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:29.702949 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:29.703420 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.703933 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.704914 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.705260 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.705336 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:29.707312 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:29.707313 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:29.707314 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:29.707322 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:29.709149 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.709659 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.709662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.709662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.709803 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.710274 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.710276 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.710276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.710461 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:29.710579 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:29.710740 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 08:21:29.710838 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 08:21:29.711382 18247 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 08:21:29 dist-test-slave-d3n8 krb5kdc[18016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
Jun 19 08:21:29 dist-test-slave-d3n8 krb5kdc[18016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857289, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.188@KRBTEST.COM for kudu/127.4.151.188@KRBTEST.COM
I20260619 08:21:29.717351 18100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.706489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34106 (local address 127.4.151.190:33791)
0619 08:21:29.706607 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:29.706609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:29.706616 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:21:29.706640 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:29.706641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:29.706656 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:29.706734 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:29.707447 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.707931 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.709036 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.709369 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.709487 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:29.713115 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:29.713117 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:29.713117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:29.713124 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:29.715123 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.716184 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.716185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.716186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.716256 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.716755 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.716756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.716756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.716906 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:29.717111 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:29.717158 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 08:21:29.717219 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260619 08:21:29.720069 18249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.711722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:44523 (local address 127.0.0.1:44384)
0619 08:21:29.711783 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.711795 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:29.711840 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:29.711973 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:29.711974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:29.712137 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:29.712178 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:29.712180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.713432 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.713433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:29.713916 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:29.713917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.714320 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.714320 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:29.714368 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:29.714756 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:29.714767 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:29.716069 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:29.718206 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:29.718207 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:29.718207 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:29.718446 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:29.718784 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:29.718784 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:29.718784 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:29.718846 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:29.719676 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:29.719677 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:29.719735 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 08:21:29.719823 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 08:21:29.719932 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 08:21:29.720079 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.711758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44384 (local address 127.4.151.188:44523)
0619 08:21:29.711815 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:29.711817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:29.711827 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:21:29.711907 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:29.711907 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:29.711917 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:29.711966 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:29.712434 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.713294 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.714028 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.714233 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.714320 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:29.716236 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:29.716237 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:29.716238 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:29.716244 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:29.718127 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.718524 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.718525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.718526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.718695 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.719057 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.719060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.719061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.719305 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:29.719625 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:29.719871 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 08:21:29.719986 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":38}
I20260619 08:21:29.722066 18247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac03e24ac2c04c0fb57a44795f3b9032: Bootstrap starting.
I20260619 08:21:29.722960 18247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac03e24ac2c04c0fb57a44795f3b9032: Neither blocks nor log segments found. Creating new log.
I20260619 08:21:29.723272 18247 log.cc:824] T 00000000000000000000000000000000 P ac03e24ac2c04c0fb57a44795f3b9032: Log is configured to *not* fsync() on all Append() calls
Jun 19 08:21:29 dist-test-slave-d3n8 krb5kdc[18016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.189@KRBTEST.COM
I20260619 08:21:29.724318 18247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac03e24ac2c04c0fb57a44795f3b9032: No bootstrap required, opened a new log
I20260619 08:21:29.726641 18247 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac03e24ac2c04c0fb57a44795f3b9032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c476ed181c41888d52f69d62a50784" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33791 } } peers { permanent_uuid: "b6647a2cc6eb45e8bff2fad5f3fa68b7" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 } } peers { permanent_uuid: "ac03e24ac2c04c0fb57a44795f3b9032" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } }
I20260619 08:21:29.726800 18247 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac03e24ac2c04c0fb57a44795f3b9032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:21:29.726863 18247 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac03e24ac2c04c0fb57a44795f3b9032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac03e24ac2c04c0fb57a44795f3b9032, State: Initialized, Role: FOLLOWER
I20260619 08:21:29.727034 18247 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac03e24ac2c04c0fb57a44795f3b9032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c476ed181c41888d52f69d62a50784" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33791 } } peers { permanent_uuid: "b6647a2cc6eb45e8bff2fad5f3fa68b7" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 } } peers { permanent_uuid: "ac03e24ac2c04c0fb57a44795f3b9032" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } }
I20260619 08:21:29.727350 18253 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac03e24ac2c04c0fb57a44795f3b9032 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c476ed181c41888d52f69d62a50784" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33791 } } peers { permanent_uuid: "b6647a2cc6eb45e8bff2fad5f3fa68b7" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 } } peers { permanent_uuid: "ac03e24ac2c04c0fb57a44795f3b9032" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } } }
I20260619 08:21:29.727422 18247 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac03e24ac2c04c0fb57a44795f3b9032 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:21:29.727521 18253 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac03e24ac2c04c0fb57a44795f3b9032 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:21:29.727916 18176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.718269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.4.151.189:34817)
0619 08:21:29.718342 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:29.718344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:29.718353 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:21:29.718375 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:29.718376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:29.718388 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:29.718450 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:29.719705 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.720406 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.721116 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.721355 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.721454 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:29.724001 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:29.724003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:29.724003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:29.724012 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:29.726432 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.726874 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.726875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.726875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.726949 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.727350 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.727351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.727352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.727482 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:29.727686 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:29.727723 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 08:21:29.727813 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 08:21:29.729595 18264 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac03e24ac2c04c0fb57a44795f3b9032: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:21:29.729657 18264 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:21:29.729771 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:21:29.730062 18179 master_runner.cc:186] Error getting master registration for 127.4.151.190:33791: OK, instance_id { permanent_uuid: "84c476ed181c41888d52f69d62a50784" instance_seqno: 1781857289326073 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 08:21:29.733191 18179 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 08:21:29.734336 18172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } attempt: 3
I20260619 08:21:29.739913 18100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.731439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.4.151.190:33791)
0619 08:21:29.731482 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:29.731483 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:29.731496 (+ 13us) server_negotiation.cc:407] Connection header received
0619 08:21:29.731596 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:29.731597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:29.731610 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:29.731642 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:29.732550 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.733284 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.734420 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.734708 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.734748 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:29.736483 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:29.736485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:29.736485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:29.736492 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:29.738505 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.738994 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.738995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.738996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.739055 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.739299 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.739300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.739300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.739430 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:29.739515 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:29.739725 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 08:21:29.739818 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 08:21:29.739913 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.731707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44390 (local address 127.4.151.188:44523)
0619 08:21:29.731746 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:29.731748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:29.732138 (+ 390us) server_negotiation.cc:407] Connection header received
0619 08:21:29.732229 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:29.732230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:29.732239 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:29.732269 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:29.733024 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.733685 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.734446 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.734753 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.734800 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:29.736277 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:29.736278 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:29.736279 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:29.736285 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:29.738426 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.738944 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.738945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.738946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.739008 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.739211 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.739212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.739212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.739406 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:29.739484 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:29.739701 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 08:21:29.739807 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Jun 19 08:21:29 dist-test-slave-d3n8 krb5kdc[18016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857289, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.189@KRBTEST.COM for kudu/127.4.151.188@KRBTEST.COM
I20260619 08:21:29.745738 18176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.731886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38262 (local address 127.4.151.189:34817)
0619 08:21:29.731924 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:29.731926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:29.737587 (+ 5661us) server_negotiation.cc:407] Connection header received
0619 08:21:29.737706 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:29.737707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:29.737720 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:29.737784 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:29.738489 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.739318 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.740451 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.740817 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.740920 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:29.742487 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:29.742488 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:29.742488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:29.742495 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:29.744276 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.744729 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.744731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.744731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.744793 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.745053 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.745054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.745055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.745195 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:29.745350 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:29.745505 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 08:21:29.745608 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:21:29.748251 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.734628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44392 (local address 127.4.151.188:44523)
0619 08:21:29.740006 (+ 5378us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:29.740008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:29.740017 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:21:29.740037 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:29.740038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:29.740048 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:29.740115 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:29.740503 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.741303 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.742338 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.742602 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.742674 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:29.745012 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:29.745013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:29.745014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:29.745023 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:29.746896 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.747317 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.747318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.747318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.747377 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.747635 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.747635 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.747636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.747778 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:29.747874 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:29.748052 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 08:21:29.748142 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5362,"thread_start_us":82,"threads_started":1}
I20260619 08:21:29.748248 18174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.734574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:44523 (local address 127.0.0.1:44392)
0619 08:21:29.734631 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.734640 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:29.734675 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:29.740131 (+ 5456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:29.740132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:29.740326 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:29.740397 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:29.740399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.741469 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.741470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:29.742265 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:29.742269 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.742705 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.742707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:29.742789 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:29.743234 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:29.743240 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:29.744912 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:29.747011 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:29.747012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:29.747013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:29.747201 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:29.747477 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:29.747478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:29.747479 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:29.747544 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:29.747879 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:29.747880 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:29.747928 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 08:21:29.748024 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 08:21:29.748139 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 08:21:29.750216 18172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6647a2cc6eb45e8bff2fad5f3fa68b7: Bootstrap starting.
I20260619 08:21:29.750929 18172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6647a2cc6eb45e8bff2fad5f3fa68b7: Neither blocks nor log segments found. Creating new log.
I20260619 08:21:29.751274 18172 log.cc:824] T 00000000000000000000000000000000 P b6647a2cc6eb45e8bff2fad5f3fa68b7: Log is configured to *not* fsync() on all Append() calls
I20260619 08:21:29.752357 18172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6647a2cc6eb45e8bff2fad5f3fa68b7: No bootstrap required, opened a new log
I20260619 08:21:29.754354 18172 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6647a2cc6eb45e8bff2fad5f3fa68b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c476ed181c41888d52f69d62a50784" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33791 } } peers { permanent_uuid: "b6647a2cc6eb45e8bff2fad5f3fa68b7" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 } } peers { permanent_uuid: "ac03e24ac2c04c0fb57a44795f3b9032" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } }
I20260619 08:21:29.754511 18172 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6647a2cc6eb45e8bff2fad5f3fa68b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:21:29.754573 18172 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6647a2cc6eb45e8bff2fad5f3fa68b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6647a2cc6eb45e8bff2fad5f3fa68b7, State: Initialized, Role: FOLLOWER
I20260619 08:21:29.754725 18172 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6647a2cc6eb45e8bff2fad5f3fa68b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c476ed181c41888d52f69d62a50784" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33791 } } peers { permanent_uuid: "b6647a2cc6eb45e8bff2fad5f3fa68b7" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 } } peers { permanent_uuid: "ac03e24ac2c04c0fb57a44795f3b9032" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } }
I20260619 08:21:29.755021 18275 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6647a2cc6eb45e8bff2fad5f3fa68b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c476ed181c41888d52f69d62a50784" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33791 } } peers { permanent_uuid: "b6647a2cc6eb45e8bff2fad5f3fa68b7" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 } } peers { permanent_uuid: "ac03e24ac2c04c0fb57a44795f3b9032" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } } }
I20260619 08:21:29.755093 18172 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6647a2cc6eb45e8bff2fad5f3fa68b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:21:29.755138 18275 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6647a2cc6eb45e8bff2fad5f3fa68b7 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 08:21:29.756937 18286 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b6647a2cc6eb45e8bff2fad5f3fa68b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:21:29.757041 18286 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:21:29.757112 18104 master_runner.cc:186] Error getting master registration for 127.4.151.190:33791: OK, instance_id { permanent_uuid: "84c476ed181c41888d52f69d62a50784" instance_seqno: 1781857289326073 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 08:21:29.759454 18104 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 08:21:29.774324 18097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } attempt: 3
Jun 19 08:21:29 dist-test-slave-d3n8 krb5kdc[18016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857289, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for kudu/127.4.151.188@KRBTEST.COM
I20260619 08:21:29.783664 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.774704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:44523 (local address 127.0.0.1:44408)
0619 08:21:29.774772 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:29.774785 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:29.774827 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:29.774995 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:29.774996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:29.775207 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:29.775266 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:29.775268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.776261 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.776263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:29.777111 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:29.777113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.777610 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.777611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:29.777673 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:29.778462 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:29.778467 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:29.780261 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:29.782454 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:29.782456 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:29.782456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:29.782708 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:29.782926 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:29.782927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:29.782927 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:29.782996 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:29.783321 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:29.783322 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:29.783372 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 08:21:29.783449 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 08:21:29.783557 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 08:21:29.783694 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:29.774813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44408 (local address 127.4.151.188:44523)
0619 08:21:29.774858 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:29.774860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:29.774870 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:21:29.774896 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:29.774897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:29.774907 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:29.774975 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:29.775357 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.776156 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.777201 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:29.777533 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:29.777608 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:29.780357 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:29.780359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:29.780360 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:29.780369 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:29.782299 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.782774 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.782775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.782775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.782841 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:29.783070 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:29.783071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:29.783072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:29.783227 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:29.783318 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:29.783493 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 08:21:29.783577 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 08:21:29.785928 18097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784: Bootstrap starting.
I20260619 08:21:29.786705 18097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784: Neither blocks nor log segments found. Creating new log.
I20260619 08:21:29.787026 18097 log.cc:824] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784: Log is configured to *not* fsync() on all Append() calls
I20260619 08:21:29.787947 18097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784: No bootstrap required, opened a new log
I20260619 08:21:29.790033 18097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c476ed181c41888d52f69d62a50784" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33791 } } peers { permanent_uuid: "b6647a2cc6eb45e8bff2fad5f3fa68b7" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 } } peers { permanent_uuid: "ac03e24ac2c04c0fb57a44795f3b9032" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } }
I20260619 08:21:29.790196 18097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:21:29.790230 18097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c476ed181c41888d52f69d62a50784, State: Initialized, Role: FOLLOWER
I20260619 08:21:29.790352 18097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c476ed181c41888d52f69d62a50784" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33791 } } peers { permanent_uuid: "b6647a2cc6eb45e8bff2fad5f3fa68b7" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 } } peers { permanent_uuid: "ac03e24ac2c04c0fb57a44795f3b9032" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } }
I20260619 08:21:29.790670 18097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 08:21:29.792869 18300 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:21:29.793045 18300 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:21:29.795197 18029 master_runner.cc:263] No leader master found from master 84c476ed181c41888d52f69d62a50784
I20260619 08:21:29.795254 18029 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 08:21:29.790644 18289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c476ed181c41888d52f69d62a50784" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33791 } } peers { permanent_uuid: "b6647a2cc6eb45e8bff2fad5f3fa68b7" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 } } peers { permanent_uuid: "ac03e24ac2c04c0fb57a44795f3b9032" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } } }
I20260619 08:21:29.796170 18289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 08:21:29.823812 18050 tablet.cc:2369] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:21:29.929906 18200 tablet.cc:2369] T 00000000000000000000000000000000 P ac03e24ac2c04c0fb57a44795f3b9032: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:21:29.994148 18125 tablet.cc:2369] T 00000000000000000000000000000000 P b6647a2cc6eb45e8bff2fad5f3fa68b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:21:30.065603 18289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:21:30.065739 18289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c476ed181c41888d52f69d62a50784" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33791 } } peers { permanent_uuid: "b6647a2cc6eb45e8bff2fad5f3fa68b7" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 } } peers { permanent_uuid: "ac03e24ac2c04c0fb57a44795f3b9032" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } }
I20260619 08:21:30.066104 18289 leader_election.cc:288] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6647a2cc6eb45e8bff2fad5f3fa68b7 (127.4.151.189:34817), ac03e24ac2c04c0fb57a44795f3b9032 (127.4.151.188:44523)
I20260619 08:21:30.066433 18218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84c476ed181c41888d52f69d62a50784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac03e24ac2c04c0fb57a44795f3b9032" is_pre_election: true
I20260619 08:21:30.066457 18146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84c476ed181c41888d52f69d62a50784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6647a2cc6eb45e8bff2fad5f3fa68b7" is_pre_election: true
I20260619 08:21:30.066612 18218 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac03e24ac2c04c0fb57a44795f3b9032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c476ed181c41888d52f69d62a50784 in term 0.
I20260619 08:21:30.066645 18146 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b6647a2cc6eb45e8bff2fad5f3fa68b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c476ed181c41888d52f69d62a50784 in term 0.
I20260619 08:21:30.066829 18047 leader_election.cc:302] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c476ed181c41888d52f69d62a50784, ac03e24ac2c04c0fb57a44795f3b9032; no voters:
I20260619 08:21:30.066974 18289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:21:30.067055 18289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:21:30.067091 18289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:21:30.067850 18289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c476ed181c41888d52f69d62a50784" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33791 } } peers { permanent_uuid: "b6647a2cc6eb45e8bff2fad5f3fa68b7" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 } } peers { permanent_uuid: "ac03e24ac2c04c0fb57a44795f3b9032" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } }
I20260619 08:21:30.068011 18289 leader_election.cc:288] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [CANDIDATE]: Term 1 election: Requested vote from peers b6647a2cc6eb45e8bff2fad5f3fa68b7 (127.4.151.189:34817), ac03e24ac2c04c0fb57a44795f3b9032 (127.4.151.188:44523)
I20260619 08:21:30.068151 18146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84c476ed181c41888d52f69d62a50784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6647a2cc6eb45e8bff2fad5f3fa68b7"
I20260619 08:21:30.068264 18146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6647a2cc6eb45e8bff2fad5f3fa68b7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:21:30.068390 18218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84c476ed181c41888d52f69d62a50784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac03e24ac2c04c0fb57a44795f3b9032"
I20260619 08:21:30.068481 18218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac03e24ac2c04c0fb57a44795f3b9032 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:21:30.069068 18146 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b6647a2cc6eb45e8bff2fad5f3fa68b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c476ed181c41888d52f69d62a50784 in term 1.
I20260619 08:21:30.069242 18048 leader_election.cc:302] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c476ed181c41888d52f69d62a50784, b6647a2cc6eb45e8bff2fad5f3fa68b7; no voters:
I20260619 08:21:30.069339 18218 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac03e24ac2c04c0fb57a44795f3b9032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c476ed181c41888d52f69d62a50784 in term 1.
I20260619 08:21:30.069375 18289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:21:30.069509 18289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [term 1 LEADER]: Becoming Leader. State: Replica: 84c476ed181c41888d52f69d62a50784, State: Running, Role: LEADER
I20260619 08:21:30.069607 18289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c476ed181c41888d52f69d62a50784" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33791 } } peers { permanent_uuid: "b6647a2cc6eb45e8bff2fad5f3fa68b7" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 } } peers { permanent_uuid: "ac03e24ac2c04c0fb57a44795f3b9032" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } }
I20260619 08:21:30.070107 18302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84c476ed181c41888d52f69d62a50784. Latest consensus state: current_term: 1 leader_uuid: "84c476ed181c41888d52f69d62a50784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c476ed181c41888d52f69d62a50784" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33791 } } peers { permanent_uuid: "b6647a2cc6eb45e8bff2fad5f3fa68b7" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 } } peers { permanent_uuid: "ac03e24ac2c04c0fb57a44795f3b9032" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } } }
I20260619 08:21:30.070191 18302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [sys.catalog]: This master's current role is: LEADER
I20260619 08:21:30.070405 18304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:21:30.070636 18304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:21:30.072454 18218 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ac03e24ac2c04c0fb57a44795f3b9032 [term 1 FOLLOWER]: Refusing update from remote peer 84c476ed181c41888d52f69d62a50784: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:21:30.072527 18146 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b6647a2cc6eb45e8bff2fad5f3fa68b7 [term 1 FOLLOWER]: Refusing update from remote peer 84c476ed181c41888d52f69d62a50784: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:21:30.072799 18302 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac03e24ac2c04c0fb57a44795f3b9032" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:21:30.072922 18289 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6647a2cc6eb45e8bff2fad5f3fa68b7" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:21:30.074254 18275 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6647a2cc6eb45e8bff2fad5f3fa68b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84c476ed181c41888d52f69d62a50784. Latest consensus state: current_term: 1 leader_uuid: "84c476ed181c41888d52f69d62a50784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c476ed181c41888d52f69d62a50784" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33791 } } peers { permanent_uuid: "b6647a2cc6eb45e8bff2fad5f3fa68b7" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 } } peers { permanent_uuid: "ac03e24ac2c04c0fb57a44795f3b9032" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } } }
I20260619 08:21:30.074361 18275 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6647a2cc6eb45e8bff2fad5f3fa68b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:21:30.074501 18253 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac03e24ac2c04c0fb57a44795f3b9032 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84c476ed181c41888d52f69d62a50784. Latest consensus state: current_term: 1 leader_uuid: "84c476ed181c41888d52f69d62a50784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c476ed181c41888d52f69d62a50784" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33791 } } peers { permanent_uuid: "b6647a2cc6eb45e8bff2fad5f3fa68b7" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 } } peers { permanent_uuid: "ac03e24ac2c04c0fb57a44795f3b9032" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } } }
I20260619 08:21:30.074621 18253 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac03e24ac2c04c0fb57a44795f3b9032 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:21:30.075465 18302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "84c476ed181c41888d52f69d62a50784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c476ed181c41888d52f69d62a50784" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33791 } } peers { permanent_uuid: "b6647a2cc6eb45e8bff2fad5f3fa68b7" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 } } peers { permanent_uuid: "ac03e24ac2c04c0fb57a44795f3b9032" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } } }
I20260619 08:21:30.075569 18302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [sys.catalog]: This master's current role is: LEADER
I20260619 08:21:30.075570 18307 mvcc.cc:205] Tried to move back new op lower bound from 7298487460133273600 to 7298487460125855744. Current Snapshot: MvccSnapshot[applied={T|T < 7298487460133273600}]
I20260619 08:21:30.075665 18275 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6647a2cc6eb45e8bff2fad5f3fa68b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "84c476ed181c41888d52f69d62a50784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c476ed181c41888d52f69d62a50784" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33791 } } peers { permanent_uuid: "b6647a2cc6eb45e8bff2fad5f3fa68b7" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 } } peers { permanent_uuid: "ac03e24ac2c04c0fb57a44795f3b9032" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } } }
I20260619 08:21:30.075737 18275 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6647a2cc6eb45e8bff2fad5f3fa68b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:21:30.075781 18302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "84c476ed181c41888d52f69d62a50784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c476ed181c41888d52f69d62a50784" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33791 } } peers { permanent_uuid: "b6647a2cc6eb45e8bff2fad5f3fa68b7" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 } } peers { permanent_uuid: "ac03e24ac2c04c0fb57a44795f3b9032" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } } }
I20260619 08:21:30.075850 18302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784 [sys.catalog]: This master's current role is: LEADER
I20260619 08:21:30.075992 18304 catalog_manager.cc:1347] Generated new cluster ID: 205064df863c4b318d0850f5c7fd082d
I20260619 08:21:30.076081 18304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:21:30.076237 18253 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac03e24ac2c04c0fb57a44795f3b9032 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "84c476ed181c41888d52f69d62a50784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c476ed181c41888d52f69d62a50784" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33791 } } peers { permanent_uuid: "b6647a2cc6eb45e8bff2fad5f3fa68b7" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 34817 } } peers { permanent_uuid: "ac03e24ac2c04c0fb57a44795f3b9032" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 44523 } } }
I20260619 08:21:30.076315 18253 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac03e24ac2c04c0fb57a44795f3b9032 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:21:30.076395 18306 mvcc.cc:205] Tried to move back new op lower bound from 7298487460133273600 to 7298487460125855744. Current Snapshot: MvccSnapshot[applied={T|T < 7298487460133273600}]
I20260619 08:21:30.088119 18304 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:21:30.088759 18304 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:21:30.091981 18304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84c476ed181c41888d52f69d62a50784: Generated new TSK 0
I20260619 08:21:30.092240 18304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:21:30.730211 18264 catalog_manager.cc:1259] Loaded cluster ID: 205064df863c4b318d0850f5c7fd082d
I20260619 08:21:30.730290 18264 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ac03e24ac2c04c0fb57a44795f3b9032: loading cluster ID for follower catalog manager: success
I20260619 08:21:30.731290 18264 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ac03e24ac2c04c0fb57a44795f3b9032: acquiring CA information for follower catalog manager: success
I20260619 08:21:30.731566 18264 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ac03e24ac2c04c0fb57a44795f3b9032: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 08:21:30.757663 18286 catalog_manager.cc:1259] Loaded cluster ID: 205064df863c4b318d0850f5c7fd082d
I20260619 08:21:30.757751 18286 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b6647a2cc6eb45e8bff2fad5f3fa68b7: loading cluster ID for follower catalog manager: success
I20260619 08:21:30.758670 18286 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b6647a2cc6eb45e8bff2fad5f3fa68b7: acquiring CA information for follower catalog manager: success
I20260619 08:21:30.758981 18286 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b6647a2cc6eb45e8bff2fad5f3fa68b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 08:21:31.155447 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 18029
I20260619 08:21:31.161854 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 18104
I20260619 08:21:31.167068 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 18179
2026-06-19T08:21:31Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3195 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:21:31 dist-test-slave-d3n8 krb5kdc[18321](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:21:31 dist-test-slave-d3n8 krb5kdc[18321](info): set up 2 sockets
Jun 19 08:21:31 dist-test-slave-d3n8 krb5kdc[18321](info): commencing operation
krb5kdc: starting...
W20260619 08:21:32.257710 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:21:32 dist-test-slave-d3n8 krb5kdc[18321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:21:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:21:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:21:32.385464 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:42443
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:45601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:42443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:21:32.467953 18334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:32.468248 18334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:32.468293 18334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:32.469542 18334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:21:32.469625 18334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:32.469650 18334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:32.469677 18334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:21:32.469707 18334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:21:32.471241 18334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:45601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:42443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:42443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:32.471542 18334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:32.471786 18334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:32.477027 18340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:32.477116 18334 server_base.cc:1034] running on GCE node
W20260619 08:21:32.477027 18339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:32.477144 18342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:32.477571 18334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:32.477890 18334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:32.479104 18334 hybrid_clock.cc:648] HybridClock initialized: now 1781857292479088 us; error 54 us; skew 500 ppm
Jun 19 08:21:32 dist-test-slave-d3n8 krb5kdc[18321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857292, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:32.485832 18334 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:21:32.486375 18334 webserver.cc:460] Webserver started at http://127.4.151.190:45701/ using document root <none> and password file <none>
I20260619 08:21:32.486636 18334 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:32.486711 18334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:21:32.486863 18334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:21:32.488030 18334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "5a0c6918031049e3af77f71c27c0498b"
format_stamp: "Formatted at 2026-06-19 08:21:32 on dist-test-slave-d3n8"
server_key: "c599a166a038e88dc310a05ccfeae01e"
server_key_iv: "54419f5c5c3d5759aecd4f8f9ba24b4a"
server_key_version: "encryptionkey@0"
I20260619 08:21:32.488413 18334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "5a0c6918031049e3af77f71c27c0498b"
format_stamp: "Formatted at 2026-06-19 08:21:32 on dist-test-slave-d3n8"
server_key: "c599a166a038e88dc310a05ccfeae01e"
server_key_iv: "54419f5c5c3d5759aecd4f8f9ba24b4a"
server_key_version: "encryptionkey@0"
I20260619 08:21:32.490039 18334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:21:32.490962 18349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:32.491183 18334 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:21:32.491310 18334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "5a0c6918031049e3af77f71c27c0498b"
format_stamp: "Formatted at 2026-06-19 08:21:32 on dist-test-slave-d3n8"
server_key: "c599a166a038e88dc310a05ccfeae01e"
server_key_iv: "54419f5c5c3d5759aecd4f8f9ba24b4a"
server_key_version: "encryptionkey@0"
I20260619 08:21:32.491400 18334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:32.505051 18334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:32.507745 18334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:32.507930 18334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:32.512533 18334 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:42443
I20260619 08:21:32.512574 18401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:42443 every 8 connection(s)
I20260619 08:21:32.512970 18334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:21:32.513960 18402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:21:32.516410 18402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a0c6918031049e3af77f71c27c0498b: Bootstrap starting.
I20260619 08:21:32.517004 18402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a0c6918031049e3af77f71c27c0498b: Neither blocks nor log segments found. Creating new log.
I20260619 08:21:32.517315 18402 log.cc:824] T 00000000000000000000000000000000 P 5a0c6918031049e3af77f71c27c0498b: Log is configured to *not* fsync() on all Append() calls
I20260619 08:21:32.518222 18402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a0c6918031049e3af77f71c27c0498b: No bootstrap required, opened a new log
I20260619 08:21:32.519960 18402 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a0c6918031049e3af77f71c27c0498b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0c6918031049e3af77f71c27c0498b" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42443 } }
I20260619 08:21:32.520104 18402 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a0c6918031049e3af77f71c27c0498b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:21:32.520128 18402 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a0c6918031049e3af77f71c27c0498b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a0c6918031049e3af77f71c27c0498b, State: Initialized, Role: FOLLOWER
I20260619 08:21:32.520309 18402 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a0c6918031049e3af77f71c27c0498b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0c6918031049e3af77f71c27c0498b" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42443 } }
I20260619 08:21:32.520392 18402 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a0c6918031049e3af77f71c27c0498b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:21:32.520416 18402 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a0c6918031049e3af77f71c27c0498b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:21:32.520469 18402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a0c6918031049e3af77f71c27c0498b [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:21:32.520792 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 18334
I20260619 08:21:32.520915 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:21:32.521130 4702 external_mini_cluster.cc:1428] Setting key efb38b4c8a12c2a7e93a8a76e5c0ca34
I20260619 08:21:32.521162 18402 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a0c6918031049e3af77f71c27c0498b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0c6918031049e3af77f71c27c0498b" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42443 } }
I20260619 08:21:32.521298 18402 leader_election.cc:302] T 00000000000000000000000000000000 P 5a0c6918031049e3af77f71c27c0498b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a0c6918031049e3af77f71c27c0498b; no voters:
I20260619 08:21:32.521502 18402 leader_election.cc:288] T 00000000000000000000000000000000 P 5a0c6918031049e3af77f71c27c0498b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:21:32.521530 18405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a0c6918031049e3af77f71c27c0498b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:21:32.521807 18405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a0c6918031049e3af77f71c27c0498b [term 1 LEADER]: Becoming Leader. State: Replica: 5a0c6918031049e3af77f71c27c0498b, State: Running, Role: LEADER
I20260619 08:21:32.521857 18402 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a0c6918031049e3af77f71c27c0498b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:21:32.521929 18405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a0c6918031049e3af77f71c27c0498b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0c6918031049e3af77f71c27c0498b" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42443 } }
I20260619 08:21:32.522284 18405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a0c6918031049e3af77f71c27c0498b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a0c6918031049e3af77f71c27c0498b. Latest consensus state: current_term: 1 leader_uuid: "5a0c6918031049e3af77f71c27c0498b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0c6918031049e3af77f71c27c0498b" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42443 } } }
I20260619 08:21:32.522408 18405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a0c6918031049e3af77f71c27c0498b [sys.catalog]: This master's current role is: LEADER
I20260619 08:21:32.522480 18407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a0c6918031049e3af77f71c27c0498b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a0c6918031049e3af77f71c27c0498b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0c6918031049e3af77f71c27c0498b" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42443 } } }
I20260619 08:21:32.522565 18407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a0c6918031049e3af77f71c27c0498b [sys.catalog]: This master's current role is: LEADER
I20260619 08:21:32.522678 18416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:21:32.523273 18416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:21:32.525769 18416 catalog_manager.cc:1347] Generated new cluster ID: 0b74143715514e9885dd6059fcf5ca7c
I20260619 08:21:32.525830 18416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:21:32 dist-test-slave-d3n8 krb5kdc[18321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:21:32.533242 18408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:32.521674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57810 (local address 127.4.151.190:42443)
0619 08:21:32.522186 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:32.522196 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:32.522230 (+ 34us) server_negotiation.cc:407] Connection header received
0619 08:21:32.522346 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:32.522353 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:32.522412 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:32.523395 (+ 983us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:32.523508 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:32.524809 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:32.525712 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:32.526304 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:32.526406 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:32.529318 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:32.529323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:32.529323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:32.529335 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:32.531486 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:32.531897 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:32.531900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:32.531901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:32.532038 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:32.532250 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:32.532251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:32.532251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:32.532589 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:32.532766 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:32.532846 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 08:21:32.532998 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":76,"threads_started":1}
I20260619 08:21:32.534307 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:21:32.534646 18416 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:21:32.535166 18416 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:21:32.543337 18416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a0c6918031049e3af77f71c27c0498b: Generated new TSK 0
I20260619 08:21:32.543570 18416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:21:32.588593 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 18334
2026-06-19T08:21:32Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1418 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:21:32 dist-test-slave-d3n8 krb5kdc[18429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:21:32 dist-test-slave-d3n8 krb5kdc[18429](info): set up 2 sockets
Jun 19 08:21:32 dist-test-slave-d3n8 krb5kdc[18429](info): commencing operation
krb5kdc: starting...
W20260619 08:21:33.645749 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:21:33 dist-test-slave-d3n8 krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:21:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:21:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:21:33.773218 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:33625
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:33625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:21:33.858592 18442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:33.858850 18442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:33.858886 18442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:33.860108 18442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:21:33.860188 18442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:33.860214 18442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:33.860252 18442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:21:33.860277 18442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:21:33.861984 18442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:33625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:33625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:33.862295 18442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:33.862552 18442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:33.868148 18450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:33.868315 18442 server_base.cc:1034] running on GCE node
W20260619 08:21:33.868157 18447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:33.868197 18448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:33.868649 18442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:33.868966 18442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:33.870203 18442 hybrid_clock.cc:648] HybridClock initialized: now 1781857293870177 us; error 41 us; skew 500 ppm
Jun 19 08:21:33 dist-test-slave-d3n8 krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857293, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:33.872901 18442 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:21:33.873471 18442 webserver.cc:460] Webserver started at http://127.4.151.190:40673/ using document root <none> and password file <none>
I20260619 08:21:33.873720 18442 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:33.873767 18442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:21:33.873859 18442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:21:33.874956 18442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "122c5c691eca4a27a136540a47e5261a"
format_stamp: "Formatted at 2026-06-19 08:21:33 on dist-test-slave-d3n8"
server_key: "d2e3ecd554c588c86a6d2683bdb1ce25"
server_key_iv: "337374d5b886b57363eaca9421c4b0a2"
server_key_version: "encryptionkey@0"
I20260619 08:21:33.875262 18442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "122c5c691eca4a27a136540a47e5261a"
format_stamp: "Formatted at 2026-06-19 08:21:33 on dist-test-slave-d3n8"
server_key: "d2e3ecd554c588c86a6d2683bdb1ce25"
server_key_iv: "337374d5b886b57363eaca9421c4b0a2"
server_key_version: "encryptionkey@0"
I20260619 08:21:33.876626 18442 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:21:33.877672 18457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:33.877885 18442 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:21:33.877975 18442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "122c5c691eca4a27a136540a47e5261a"
format_stamp: "Formatted at 2026-06-19 08:21:33 on dist-test-slave-d3n8"
server_key: "d2e3ecd554c588c86a6d2683bdb1ce25"
server_key_iv: "337374d5b886b57363eaca9421c4b0a2"
server_key_version: "encryptionkey@0"
I20260619 08:21:33.878036 18442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:33.910037 18442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:33.912482 18442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:33.912632 18442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:33.916893 18442 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:33625
I20260619 08:21:33.916953 18509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:33625 every 8 connection(s)
I20260619 08:21:33.917359 18442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:21:33.918732 18510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:21:33.919124 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 18442
I20260619 08:21:33.919225 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:21:33.919536 4702 external_mini_cluster.cc:1428] Setting key f8c9c6ff7eefa2e240470ca9979be40f
I20260619 08:21:33.922027 18510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a: Bootstrap starting.
I20260619 08:21:33.922794 18510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a: Neither blocks nor log segments found. Creating new log.
I20260619 08:21:33.923112 18510 log.cc:824] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a: Log is configured to *not* fsync() on all Append() calls
I20260619 08:21:33.923903 18510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a: No bootstrap required, opened a new log
I20260619 08:21:33.926230 18510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122c5c691eca4a27a136540a47e5261a" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33625 } }
I20260619 08:21:33.926374 18510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:21:33.926396 18510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122c5c691eca4a27a136540a47e5261a, State: Initialized, Role: FOLLOWER
I20260619 08:21:33.926539 18510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122c5c691eca4a27a136540a47e5261a" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33625 } }
I20260619 08:21:33.926623 18510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:21:33.926646 18510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:21:33.926734 18510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:21:33.927348 18510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122c5c691eca4a27a136540a47e5261a" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33625 } }
Jun 19 08:21:33 dist-test-slave-d3n8 krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:21:33.927479 18510 leader_election.cc:302] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 122c5c691eca4a27a136540a47e5261a; no voters:
I20260619 08:21:33.927672 18510 leader_election.cc:288] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:21:33.927709 18515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:21:33.927830 18515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [term 1 LEADER]: Becoming Leader. State: Replica: 122c5c691eca4a27a136540a47e5261a, State: Running, Role: LEADER
I20260619 08:21:33.927963 18510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:21:33.927965 18515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122c5c691eca4a27a136540a47e5261a" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33625 } }
I20260619 08:21:33.928779 18517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 122c5c691eca4a27a136540a47e5261a. Latest consensus state: current_term: 1 leader_uuid: "122c5c691eca4a27a136540a47e5261a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122c5c691eca4a27a136540a47e5261a" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33625 } } }
I20260619 08:21:33.928867 18517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [sys.catalog]: This master's current role is: LEADER
I20260619 08:21:33.929423 18516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "122c5c691eca4a27a136540a47e5261a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122c5c691eca4a27a136540a47e5261a" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33625 } } }
I20260619 08:21:33.930308 18516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [sys.catalog]: This master's current role is: LEADER
W20260619 08:21:33.929906 18531 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:21:33.930380 18531 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:21:33.930447 18529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:21:33.930634 18529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:21:33.931907 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:33.920254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46446 (local address 127.4.151.190:33625)
0619 08:21:33.920470 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:33.920480 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:33.920515 (+ 35us) server_negotiation.cc:407] Connection header received
0619 08:21:33.921048 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:33.921056 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:33.921117 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:33.921293 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:33.922031 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:33.923451 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:33.924402 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:33.925023 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:33.925148 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:33.927966 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:33.927970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:33.927970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:33.927984 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:33.930132 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:33.930596 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:33.930600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:33.930600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:33.930720 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:33.930947 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:33.930948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:33.930948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:33.931188 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:33.931366 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:33.931494 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 08:21:33.931637 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":95,"threads_started":1}
I20260619 08:21:33.932451 18529 catalog_manager.cc:1347] Generated new cluster ID: 8a85f3056c48428ea220d6bcebaa75ca
I20260619 08:21:33.932515 18529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:21:33.966607 18529 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:21:33.967254 18529 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:21:33.973268 18529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a: Generated new TSK 0
I20260619 08:21:33.973588 18529 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.151.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
I20260619 08:21:34.009951 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:33625
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:21:34.098551 18536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:34.098773 18536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:34.098800 18536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:34.099947 18536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:34.099988 18536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:34.100015 18536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:21:34.101459 18536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:33625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:34.101660 18536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:34.101845 18536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:34.107458 18542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:34.107595 18541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:34.107654 18544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:34.107478 18536 server_base.cc:1034] running on GCE node
I20260619 08:21:34.107980 18536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:34.108270 18536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:34.109472 18536 hybrid_clock.cc:648] HybridClock initialized: now 1781857294109456 us; error 31 us; skew 500 ppm
Jun 19 08:21:34 dist-test-slave-d3n8 krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857294, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:34.112159 18536 init.cc:377] Logged in from keytab as kudu/127.4.151.129@KRBTEST.COM (short username kudu)
I20260619 08:21:34.112637 18536 webserver.cc:460] Webserver started at http://127.4.151.129:44795/ using document root <none> and password file <none>
I20260619 08:21:34.112851 18536 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:34.112893 18536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:21:34.112998 18536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:21:34.114121 18536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "93af7b16aab148ec9d117f477a745284"
format_stamp: "Formatted at 2026-06-19 08:21:34 on dist-test-slave-d3n8"
server_key: "630de7b74e948fa46757e98c77d7bf23"
server_key_iv: "8819b7971ea9a1e5de7d1f546ea779fa"
server_key_version: "encryptionkey@0"
I20260619 08:21:34.114426 18536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "93af7b16aab148ec9d117f477a745284"
format_stamp: "Formatted at 2026-06-19 08:21:34 on dist-test-slave-d3n8"
server_key: "630de7b74e948fa46757e98c77d7bf23"
server_key_iv: "8819b7971ea9a1e5de7d1f546ea779fa"
server_key_version: "encryptionkey@0"
I20260619 08:21:34.115790 18536 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:21:34.116668 18551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:34.116919 18536 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:21:34.117005 18536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "93af7b16aab148ec9d117f477a745284"
format_stamp: "Formatted at 2026-06-19 08:21:34 on dist-test-slave-d3n8"
server_key: "630de7b74e948fa46757e98c77d7bf23"
server_key_iv: "8819b7971ea9a1e5de7d1f546ea779fa"
server_key_version: "encryptionkey@0"
I20260619 08:21:34.117064 18536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:34.127692 18536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:34.130342 18536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:34.130465 18536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:34.130661 18536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:21:34.131042 18536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:21:34.131079 18536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:34.131102 18536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:21:34.131155 18536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:34.137084 18536 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:33811
I20260619 08:21:34.137126 18664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:33811 every 8 connection(s)
I20260619 08:21:34.137540 18536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
Jun 19 08:21:34 dist-test-slave-d3n8 krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857294, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:21:34.145609 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 18536
I20260619 08:21:34.145744 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:21:34.146001 4702 external_mini_cluster.cc:1428] Setting key 4927cd9d64bea58e4d7dc3a65dfd9509
I20260619 08:21:34.148211 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:34.138413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:59719 (local address 127.4.151.190:33625)
0619 08:21:34.138488 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:34.138490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:34.138764 (+ 274us) server_negotiation.cc:407] Connection header received
0619 08:21:34.139285 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:34.139286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:34.139298 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:34.139346 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:34.140382 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.140913 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.142137 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.142360 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.142426 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:34.144851 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:34.144859 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:34.144859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:34.144867 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:34.146543 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:34.147030 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:34.147031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:34.147031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:34.147096 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:34.147428 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:34.147430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:34.147430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:34.147589 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:34.147665 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:34.147938 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 08:21:34.148054 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 08:21:34.148582 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:34.138375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.129:59719)
0619 08:21:34.138576 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:34.138626 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:34.139158 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:34.139419 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:34.139422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:34.140040 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:34.140276 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:34.140281 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.141007 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.141008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:34.142035 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:34.142038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.142457 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.142457 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:34.142580 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:34.143234 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:34.143243 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:34.144755 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:34.146634 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:34.146641 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:34.146643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:34.146954 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:34.147185 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:34.147185 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:34.147185 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:34.147353 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:34.147660 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:34.147665 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:34.147801 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 08:21:34.147895 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 08:21:34.148058 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":115,"threads_started":1}
I20260619 08:21:34.149179 18665 heartbeater.cc:347] Connected to a master server at 127.4.151.190:33625
I20260619 08:21:34.149304 18665 heartbeater.cc:464] Registering TS with master...
I20260619 08:21:34.149559 18665 heartbeater.cc:511] Master 127.4.151.190:33625 requested a full tablet report, sending...
I20260619 08:21:34.150101 18474 ts_manager.cc:194] Registered new tserver with Master: 93af7b16aab148ec9d117f477a745284 (127.4.151.129:33811)
I20260619 08:21:34.150717 18474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.129@KRBTEST.COM'} at 127.4.151.129:59719
WARNING: no policy specified for kudu/127.4.151.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
I20260619 08:21:34.174065 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:33625
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:21:34.260035 18670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:34.260259 18670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:34.260285 18670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:34.261641 18670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:34.261699 18670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:34.261729 18670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:21:34.263329 18670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:33625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:34.263613 18670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:34.263888 18670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:34.269523 18676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:34.269517 18678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:34.269493 18675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:34.269713 18670 server_base.cc:1034] running on GCE node
I20260619 08:21:34.269976 18670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:34.270262 18670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:34.271462 18670 hybrid_clock.cc:648] HybridClock initialized: now 1781857294271455 us; error 53 us; skew 500 ppm
Jun 19 08:21:34 dist-test-slave-d3n8 krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857294, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:34.274438 18670 init.cc:377] Logged in from keytab as kudu/127.4.151.130@KRBTEST.COM (short username kudu)
I20260619 08:21:34.274914 18670 webserver.cc:460] Webserver started at http://127.4.151.130:41893/ using document root <none> and password file <none>
I20260619 08:21:34.275177 18670 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:34.275249 18670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:21:34.275398 18670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:21:34.276561 18670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "8b2473c351fe4d4a9be83937dbbc9f5b"
format_stamp: "Formatted at 2026-06-19 08:21:34 on dist-test-slave-d3n8"
server_key: "c83a18e6fa9c6720886437006150bddb"
server_key_iv: "70fd3c6974b6d49d2c471c88d85854e3"
server_key_version: "encryptionkey@0"
I20260619 08:21:34.276947 18670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "8b2473c351fe4d4a9be83937dbbc9f5b"
format_stamp: "Formatted at 2026-06-19 08:21:34 on dist-test-slave-d3n8"
server_key: "c83a18e6fa9c6720886437006150bddb"
server_key_iv: "70fd3c6974b6d49d2c471c88d85854e3"
server_key_version: "encryptionkey@0"
I20260619 08:21:34.278446 18670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:21:34.279361 18685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:34.279558 18670 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:21:34.279662 18670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "8b2473c351fe4d4a9be83937dbbc9f5b"
format_stamp: "Formatted at 2026-06-19 08:21:34 on dist-test-slave-d3n8"
server_key: "c83a18e6fa9c6720886437006150bddb"
server_key_iv: "70fd3c6974b6d49d2c471c88d85854e3"
server_key_version: "encryptionkey@0"
I20260619 08:21:34.279745 18670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:34.287384 18670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:34.289877 18670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:34.290063 18670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:34.290383 18670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:21:34.290721 18670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:21:34.290782 18670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:34.290827 18670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:21:34.290853 18670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:34.297158 18670 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:43707
I20260619 08:21:34.297240 18798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:43707 every 8 connection(s)
I20260619 08:21:34.297654 18670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
I20260619 08:21:34.299827 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 18670
I20260619 08:21:34.299947 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:21:34.300141 4702 external_mini_cluster.cc:1428] Setting key e21032ccd0b64d0aa24e1d2a4b7a97f1
Jun 19 08:21:34 dist-test-slave-d3n8 krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857294, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:21:34.309363 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:34.298484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:48653 (local address 127.4.151.190:33625)
0619 08:21:34.298579 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:34.298582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:34.298826 (+ 244us) server_negotiation.cc:407] Connection header received
0619 08:21:34.299281 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:34.299283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:34.299295 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:34.299367 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:34.300687 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.301461 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.302920 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.303241 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.303327 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:34.305785 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:34.305787 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:34.305787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:34.305796 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:34.307550 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:34.308168 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:34.308169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:34.308169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:34.308242 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:34.308575 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:34.308576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:34.308576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:34.308748 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:34.308836 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:34.309154 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 08:21:34.309253 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 08:21:34.309494 18801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:34.298453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.130:48653)
0619 08:21:34.298622 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:34.298665 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:34.299164 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:34.299424 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:34.299428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:34.300288 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:34.300594 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:34.300601 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.301561 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.301562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:34.302804 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:34.302807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.303361 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.303362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:34.303525 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:34.304149 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:34.304158 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:34.305671 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:34.307837 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:34.307845 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:34.307847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:34.308081 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:34.308325 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:34.308326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:34.308326 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:34.308438 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:34.308845 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:34.308851 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:34.308984 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 08:21:34.309101 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 08:21:34.309247 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for kudu/127.4.151.131@KRBTEST.COM; defaulting to no policy
I20260619 08:21:34.310081 18799 heartbeater.cc:347] Connected to a master server at 127.4.151.190:33625
I20260619 08:21:34.310211 18799 heartbeater.cc:464] Registering TS with master...
I20260619 08:21:34.310398 18799 heartbeater.cc:511] Master 127.4.151.190:33625 requested a full tablet report, sending...
I20260619 08:21:34.310891 18474 ts_manager.cc:194] Registered new tserver with Master: 8b2473c351fe4d4a9be83937dbbc9f5b (127.4.151.130:43707)
I20260619 08:21:34.311290 18474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.130@KRBTEST.COM'} at 127.4.151.130:48653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
I20260619 08:21:34.330291 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:33625
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:21:34.414168 18804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:34.414389 18804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:34.414414 18804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:34.415661 18804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:34.415704 18804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:34.415731 18804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:21:34.417203 18804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:33625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:34.417485 18804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:34.417743 18804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:34.423085 18810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:34.423058 18812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:34.423221 18809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:34.423408 18804 server_base.cc:1034] running on GCE node
I20260619 08:21:34.423619 18804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:34.423902 18804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:34.425143 18804 hybrid_clock.cc:648] HybridClock initialized: now 1781857294425131 us; error 38 us; skew 500 ppm
Jun 19 08:21:34 dist-test-slave-d3n8 krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857294, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:34.428202 18804 init.cc:377] Logged in from keytab as kudu/127.4.151.131@KRBTEST.COM (short username kudu)
I20260619 08:21:34.428679 18804 webserver.cc:460] Webserver started at http://127.4.151.131:45601/ using document root <none> and password file <none>
I20260619 08:21:34.428980 18804 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:34.429030 18804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:21:34.429152 18804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:21:34.430373 18804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "d315c1374d844a9ab7774177f8150672"
format_stamp: "Formatted at 2026-06-19 08:21:34 on dist-test-slave-d3n8"
server_key: "793bdcbcef4d0dca95a2c22a17abc9c0"
server_key_iv: "ef676be221720c2b6e670f1e463e992c"
server_key_version: "encryptionkey@0"
I20260619 08:21:34.430773 18804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "d315c1374d844a9ab7774177f8150672"
format_stamp: "Formatted at 2026-06-19 08:21:34 on dist-test-slave-d3n8"
server_key: "793bdcbcef4d0dca95a2c22a17abc9c0"
server_key_iv: "ef676be221720c2b6e670f1e463e992c"
server_key_version: "encryptionkey@0"
I20260619 08:21:34.432205 18804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:21:34.432998 18819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:34.433207 18804 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:21:34.433305 18804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "d315c1374d844a9ab7774177f8150672"
format_stamp: "Formatted at 2026-06-19 08:21:34 on dist-test-slave-d3n8"
server_key: "793bdcbcef4d0dca95a2c22a17abc9c0"
server_key_iv: "ef676be221720c2b6e670f1e463e992c"
server_key_version: "encryptionkey@0"
I20260619 08:21:34.433426 18804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:34.455885 18804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:34.458353 18804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:34.458603 18804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:34.458886 18804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:21:34.459230 18804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:21:34.459295 18804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:34.459337 18804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:21:34.459367 18804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:34.465787 18804 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:33519
I20260619 08:21:34.465817 18932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:33519 every 8 connection(s)
I20260619 08:21:34.466193 18804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
Jun 19 08:21:34 dist-test-slave-d3n8 krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857294, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:21:34.476030 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 18804
I20260619 08:21:34.476171 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:21:34.476301 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:34.467075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:53665 (local address 127.4.151.190:33625)
0619 08:21:34.467177 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:34.467179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:34.467365 (+ 186us) server_negotiation.cc:407] Connection header received
0619 08:21:34.467922 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:34.467924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:34.467936 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:34.467995 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:34.469012 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.469605 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.470734 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.470931 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.471024 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:34.473436 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:34.473438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:34.473438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:34.473447 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:34.474966 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:34.475387 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:34.475388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:34.475388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:34.475439 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:34.475640 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:34.475640 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:34.475640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:34.475771 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:34.475820 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:34.476108 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 08:21:34.476204 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 08:21:34.476370 4702 external_mini_cluster.cc:1428] Setting key 5311f696c56727e0bf88e8003d81e3ea
I20260619 08:21:34.476584 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:34.467045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.131:53665)
0619 08:21:34.467239 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:34.467277 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:34.467806 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:34.468082 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:34.468085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:34.468656 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:34.468860 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:34.468865 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.469706 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.469708 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:34.470608 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:34.470610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.471033 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.471035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:34.471170 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:34.471751 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:34.471759 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:34.473208 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:34.475048 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:34.475055 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:34.475057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:34.475321 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:34.475499 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:34.475500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:34.475500 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:34.475599 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:34.475827 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:34.475831 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:34.475971 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 08:21:34.476058 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 08:21:34.476298 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
I20260619 08:21:34.477115 18933 heartbeater.cc:347] Connected to a master server at 127.4.151.190:33625
I20260619 08:21:34.477231 18933 heartbeater.cc:464] Registering TS with master...
I20260619 08:21:34.477443 18933 heartbeater.cc:511] Master 127.4.151.190:33625 requested a full tablet report, sending...
I20260619 08:21:34.477841 18474 ts_manager.cc:194] Registered new tserver with Master: d315c1374d844a9ab7774177f8150672 (127.4.151.131:33519)
I20260619 08:21:34.478233 18474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.131@KRBTEST.COM'} at 127.4.151.131:53665
I20260619 08:21:34.488476 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:21:34.496598 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:34.489980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46450 (local address 127.4.151.190:33625)
0619 08:21:34.490078 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:34.490080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:34.490249 (+ 169us) server_negotiation.cc:407] Connection header received
0619 08:21:34.490326 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:34.490326 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:34.490337 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:34.490411 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:34.491026 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.491523 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.492284 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.492476 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.492610 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:34.493954 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:34.493955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:34.493955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:34.493963 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:34.495370 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:34.495741 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:34.495743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:34.495743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:34.495801 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:34.496017 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:34.496018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:34.496019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:34.496178 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:34.496290 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:34.496397 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 08:21:34.496482 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 08:21:34.498750 18474 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:21:34.500386 18474 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:21:34.507606 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:34.503007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59586 (local address 127.4.151.131:33519)
0619 08:21:34.503230 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:34.503235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:34.503307 (+ 72us) server_negotiation.cc:407] Connection header received
0619 08:21:34.503476 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:34.503479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:34.503495 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:34.503583 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:21:34.503944 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.505069 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.506307 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.507055 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.507130 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:34.507204 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:34.507433 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 08:21:34.507500 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
I20260619 08:21:34.507647 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:34.502875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:33811 (local address 127.0.0.1:33876)
0619 08:21:34.503204 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:34.503250 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:34.503319 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:34.503633 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:34.503636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:34.503638 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:21:34.503714 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:34.503717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.505169 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.505170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:34.506338 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:34.506339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.507220 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.507221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:34.507412 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:34.507417 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:21:34.507501 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 08:21:34.507540 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":28,"thread_start_us":51,"threads_started":1}
I20260619 08:21:34.507659 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:34.502887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:33519 (local address 127.0.0.1:59586)
0619 08:21:34.503224 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:34.503250 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:34.503319 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:34.503799 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:34.503800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:34.503802 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:21:34.503856 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:34.503858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.505179 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.505180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:34.506213 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:34.506216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.507239 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.507240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:34.507315 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:34.507319 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:21:34.507412 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 08:21:34.507520 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":86,"thread_start_us":96,"threads_started":1}
I20260619 08:21:34.507835 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:34.503233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50766 (local address 127.4.151.130:43707)
0619 08:21:34.503378 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:34.503382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:34.503498 (+ 116us) server_negotiation.cc:407] Connection header received
0619 08:21:34.503550 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:34.503553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:34.503570 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:34.503641 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:21:34.503925 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.504731 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.506338 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.506948 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.507056 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:34.507120 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:34.507681 (+ 561us) server_negotiation.cc:299] Negotiation successful
0619 08:21:34.507734 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260619 08:21:34.507999 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:34.503118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:43707 (local address 127.0.0.1:50766)
0619 08:21:34.503431 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:34.503440 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:34.503489 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:34.503704 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:34.503705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:34.503708 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:21:34.503841 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:34.503843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.504858 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.504859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:34.506234 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:34.506238 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.507058 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.507060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:34.507153 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:34.507170 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 08:21:34.507391 (+ 221us) client_negotiation.cc:240] Negotiation successful
0619 08:21:34.507512 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":63,"threads_started":1}
I20260619 08:21:34.508268 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:34.503000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33876 (local address 127.4.151.129:33811)
0619 08:21:34.503418 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:34.503422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:34.503431 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:21:34.503472 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:34.503474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:34.503491 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:34.503798 (+ 307us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:21:34.503875 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.505042 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.506407 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.507095 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.507190 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:34.507243 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:34.508103 (+ 860us) server_negotiation.cc:299] Negotiation successful
0619 08:21:34.508165 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":98,"threads_started":1}
I20260619 08:21:34.508843 18733 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc92b22dabd430abc58ddcfebff2ed7 (DEFAULT_TABLE table=test-table [id=b783aa3dcdfe4a9ca78980eafd76d392]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:21:34.508879 18867 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc92b22dabd430abc58ddcfebff2ed7 (DEFAULT_TABLE table=test-table [id=b783aa3dcdfe4a9ca78980eafd76d392]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:21:34.508877 18599 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc92b22dabd430abc58ddcfebff2ed7 (DEFAULT_TABLE table=test-table [id=b783aa3dcdfe4a9ca78980eafd76d392]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:21:34.509220 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc92b22dabd430abc58ddcfebff2ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:21:34.509222 18867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc92b22dabd430abc58ddcfebff2ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:21:34.509238 18733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc92b22dabd430abc58ddcfebff2ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:21:34.512075 18950 tablet_bootstrap.cc:492] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b: Bootstrap starting.
I20260619 08:21:34.512653 18952 tablet_bootstrap.cc:492] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284: Bootstrap starting.
I20260619 08:21:34.512897 18951 tablet_bootstrap.cc:492] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672: Bootstrap starting.
I20260619 08:21:34.513161 18950 tablet_bootstrap.cc:654] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b: Neither blocks nor log segments found. Creating new log.
I20260619 08:21:34.513725 18951 tablet_bootstrap.cc:654] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672: Neither blocks nor log segments found. Creating new log.
I20260619 08:21:34.513962 18952 tablet_bootstrap.cc:654] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284: Neither blocks nor log segments found. Creating new log.
I20260619 08:21:34.513916 18950 log.cc:824] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b: Log is configured to *not* fsync() on all Append() calls
I20260619 08:21:34.514374 18951 log.cc:824] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672: Log is configured to *not* fsync() on all Append() calls
I20260619 08:21:34.514601 18952 log.cc:824] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284: Log is configured to *not* fsync() on all Append() calls
I20260619 08:21:34.514995 18950 tablet_bootstrap.cc:492] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b: No bootstrap required, opened a new log
I20260619 08:21:34.515090 18950 ts_tablet_manager.cc:1397] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 08:21:34.515166 18951 tablet_bootstrap.cc:492] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672: No bootstrap required, opened a new log
I20260619 08:21:34.515244 18951 ts_tablet_manager.cc:1397] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 08:21:34.517360 18950 raft_consensus.cc:348] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:34.517360 18951 raft_consensus.cc:348] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:34.517602 18951 raft_consensus.cc:374] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:21:34.517602 18950 raft_consensus.cc:374] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:21:34.517645 18950 raft_consensus.cc:729] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b2473c351fe4d4a9be83937dbbc9f5b, State: Initialized, Role: FOLLOWER
I20260619 08:21:34.517645 18951 raft_consensus.cc:729] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d315c1374d844a9ab7774177f8150672, State: Initialized, Role: FOLLOWER
I20260619 08:21:34.517810 18951 consensus_queue.cc:260] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:34.517838 18950 consensus_queue.cc:260] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:34.518067 18952 tablet_bootstrap.cc:492] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284: No bootstrap required, opened a new log
I20260619 08:21:34.518131 18951 ts_tablet_manager.cc:1428] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 08:21:34.518131 18950 ts_tablet_manager.cc:1428] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260619 08:21:34.518167 18952 ts_tablet_manager.cc:1397] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 08:21:34.518270 18799 heartbeater.cc:503] Master 127.4.151.190:33625 was elected leader, sending a full tablet report...
I20260619 08:21:34.518301 18933 heartbeater.cc:503] Master 127.4.151.190:33625 was elected leader, sending a full tablet report...
I20260619 08:21:34.520126 18952 raft_consensus.cc:348] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:34.520367 18952 raft_consensus.cc:374] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:21:34.520411 18952 raft_consensus.cc:729] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93af7b16aab148ec9d117f477a745284, State: Initialized, Role: FOLLOWER
I20260619 08:21:34.520574 18952 consensus_queue.cc:260] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:34.520880 18952 ts_tablet_manager.cc:1428] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:21:34.520893 18665 heartbeater.cc:503] Master 127.4.151.190:33625 was elected leader, sending a full tablet report...
W20260619 08:21:34.548285 18800 tablet.cc:2369] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:21:34.638535 18666 tablet.cc:2369] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:21:34.716840 18934 tablet.cc:2369] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:21:34.812083 18957 raft_consensus.cc:482] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:21:34.812654 18957 raft_consensus.cc:504] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:34.813153 18957 leader_election.cc:288] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93af7b16aab148ec9d117f477a745284 (127.4.151.129:33811), 8b2473c351fe4d4a9be83937dbbc9f5b (127.4.151.130:43707)
I20260619 08:21:34.817684 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:34.813596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:34931 (local address 127.4.151.129:33811)
0619 08:21:34.813664 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:34.813666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:34.814132 (+ 466us) server_negotiation.cc:407] Connection header received
0619 08:21:34.814166 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:34.814167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:34.814177 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:34.814246 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:21:34.814402 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.815060 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.816499 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.817263 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.817348 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:34.817356 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:34.817548 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 08:21:34.817595 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 08:21:34.817684 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:34.813538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:55277 (local address 127.4.151.130:43707)
0619 08:21:34.813600 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:34.813602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:34.813613 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:21:34.813644 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:34.813645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:34.813655 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:34.813767 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:21:34.814061 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.814828 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.816499 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.817179 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.817281 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:34.817292 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:34.817548 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 08:21:34.817603 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 08:21:34.817737 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:34.813467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:43707 (local address 127.4.151.131:55277)
0619 08:21:34.813517 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:34.813532 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:34.813581 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:34.813772 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:34.813773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:34.813775 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:21:34.813900 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:34.813904 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.814942 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.814943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:34.816362 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:34.816365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.817326 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.817327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:34.817447 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:34.817451 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:21:34.817522 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 08:21:34.817631 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":104}
I20260619 08:21:34.817744 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:34.813673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:33811 (local address 127.4.151.131:34931)
0619 08:21:34.813874 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:34.813885 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:34.813956 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:34.814268 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:34.814269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:34.814270 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:21:34.814327 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:34.814328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.815157 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.815158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:34.816362 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:34.816365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.817362 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.817362 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:34.817453 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:34.817455 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:21:34.817522 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 08:21:34.817629 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":14,"thread_start_us":117,"threads_started":1}
I20260619 08:21:34.818171 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc92b22dabd430abc58ddcfebff2ed7" candidate_uuid: "d315c1374d844a9ab7774177f8150672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93af7b16aab148ec9d117f477a745284" is_pre_election: true
I20260619 08:21:34.818151 18753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc92b22dabd430abc58ddcfebff2ed7" candidate_uuid: "d315c1374d844a9ab7774177f8150672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" is_pre_election: true
I20260619 08:21:34.818331 18753 raft_consensus.cc:2454] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d315c1374d844a9ab7774177f8150672 in term 0.
I20260619 08:21:34.818331 18619 raft_consensus.cc:2454] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d315c1374d844a9ab7774177f8150672 in term 0.
I20260619 08:21:34.818621 18821 leader_election.cc:302] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b2473c351fe4d4a9be83937dbbc9f5b, d315c1374d844a9ab7774177f8150672; no voters:
I20260619 08:21:34.818764 18957 raft_consensus.cc:2781] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:21:34.818818 18957 raft_consensus.cc:482] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:21:34.818841 18957 raft_consensus.cc:3037] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:21:34.819687 18957 raft_consensus.cc:504] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:34.819844 18957 leader_election.cc:288] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672 [CANDIDATE]: Term 1 election: Requested vote from peers 93af7b16aab148ec9d117f477a745284 (127.4.151.129:33811), 8b2473c351fe4d4a9be83937dbbc9f5b (127.4.151.130:43707)
I20260619 08:21:34.820008 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc92b22dabd430abc58ddcfebff2ed7" candidate_uuid: "d315c1374d844a9ab7774177f8150672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93af7b16aab148ec9d117f477a745284"
I20260619 08:21:34.820024 18753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc92b22dabd430abc58ddcfebff2ed7" candidate_uuid: "d315c1374d844a9ab7774177f8150672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b"
I20260619 08:21:34.820120 18619 raft_consensus.cc:3037] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:21:34.820114 18753 raft_consensus.cc:3037] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:21:34.820968 18619 raft_consensus.cc:2454] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d315c1374d844a9ab7774177f8150672 in term 1.
I20260619 08:21:34.820971 18753 raft_consensus.cc:2454] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d315c1374d844a9ab7774177f8150672 in term 1.
I20260619 08:21:34.821218 18822 leader_election.cc:302] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93af7b16aab148ec9d117f477a745284, d315c1374d844a9ab7774177f8150672; no voters:
I20260619 08:21:34.821355 18957 raft_consensus.cc:2781] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:21:34.821602 18957 raft_consensus.cc:686] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672 [term 1 LEADER]: Becoming Leader. State: Replica: d315c1374d844a9ab7774177f8150672, State: Running, Role: LEADER
I20260619 08:21:34.821712 18957 consensus_queue.cc:237] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:34.822667 18473 catalog_manager.cc:5495] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672 reported cstate change: term changed from 0 to 1, leader changed from <none> to d315c1374d844a9ab7774177f8150672 (127.4.151.131). New cstate: current_term: 1 leader_uuid: "d315c1374d844a9ab7774177f8150672" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } health_report { overall_health: UNKNOWN } } }
I20260619 08:21:34.864651 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:34.862164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59598 (local address 127.4.151.131:33519)
0619 08:21:34.862272 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:34.862273 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:34.862281 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:21:34.862317 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:34.862318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:34.862337 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:34.862424 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:21:34.862580 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.863067 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.863847 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:34.864042 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:34.864132 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:34.864299 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:21:34.864415 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:21:34.864518 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:34.864537 (+ 19us) server_negotiation.cc:299] Negotiation successful
0619 08:21:34.864558 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 08:21:34.866657 18753 raft_consensus.cc:1261] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b [term 1 FOLLOWER]: Refusing update from remote peer d315c1374d844a9ab7774177f8150672: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:21:34.866664 18619 raft_consensus.cc:1261] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Refusing update from remote peer d315c1374d844a9ab7774177f8150672: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:21:34.867053 18957 consensus_queue.cc:1035] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:21:34.867228 18960 consensus_queue.cc:1035] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:21:34.869663 18963 mvcc.cc:205] Tried to move back new op lower bound from 7298487479770841088 to 7298487479590416384. Current Snapshot: MvccSnapshot[applied={T|T < 7298487479770841088}]
I20260619 08:21:34.869874 18965 mvcc.cc:205] Tried to move back new op lower bound from 7298487479770841088 to 7298487479590416384. Current Snapshot: MvccSnapshot[applied={T|T < 7298487479770841088}]
I20260619 08:21:34.872243 18474 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:21:34.872355 18474 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:21:34.873136 18474 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a: Sending DeleteTablet for 3 replicas of tablet 3fc92b22dabd430abc58ddcfebff2ed7
I20260619 08:21:34.873533 18599 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fc92b22dabd430abc58ddcfebff2ed7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:21:34 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:33876
I20260619 08:21:34.873811 18973 tablet_replica.cc:330] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284: stopping tablet replica
I20260619 08:21:34.873821 18733 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fc92b22dabd430abc58ddcfebff2ed7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:21:34 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:50766
I20260619 08:21:34.873934 18973 raft_consensus.cc:2229] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:21:34.874111 18973 raft_consensus.cc:2258] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:21:34.874105 18867 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fc92b22dabd430abc58ddcfebff2ed7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:21:34 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:59586
I20260619 08:21:34.874140 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 18536
I20260619 08:21:34.875510 18975 tablet_replica.cc:330] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672: stopping tablet replica
I20260619 08:21:34.875628 18975 raft_consensus.cc:2229] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:21:34.875780 18975 raft_consensus.cc:2258] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:21:34.876274 18975 ts_tablet_manager.cc:1905] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:21:34.878479 18975 ts_tablet_manager.cc:1918] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:21:34.878566 18975 log.cc:1196] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/3fc92b22dabd430abc58ddcfebff2ed7
I20260619 08:21:34.878844 18975 ts_tablet_manager.cc:1939] T 3fc92b22dabd430abc58ddcfebff2ed7 P d315c1374d844a9ab7774177f8150672: Deleting consensus metadata
I20260619 08:21:34.879347 18459 catalog_manager.cc:4841] TS d315c1374d844a9ab7774177f8150672 (127.4.151.131:33519): tablet 3fc92b22dabd430abc58ddcfebff2ed7 (table test-table [id=b783aa3dcdfe4a9ca78980eafd76d392]) successfully deleted
W20260619 08:21:34.880395 18460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.151.129:33811 (error 108)
I20260619 08:21:34.880990 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:34.880848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:33811 (local address 127.0.0.1:33888)
0619 08:21:34.880908 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:34.880954 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.129:33811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 08:21:34.881093 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 18670
I20260619 08:21:34.881098 18974 tablet_replica.cc:330] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b: stopping tablet replica
W20260619 08:21:34.886202 18460 catalog_manager.cc:4584] TS 93af7b16aab148ec9d117f477a745284 (127.4.151.129:33811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3fc92b22dabd430abc58ddcfebff2ed7: Network error: Client connection negotiation failed: client connection to 127.4.151.129:33811: connect: Connection refused (error 111)
I20260619 08:21:34.887499 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 18804
I20260619 08:21:34.887741 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:34.887577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:43707 (local address 127.0.0.1:50776)
0619 08:21:34.887635 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:34.887698 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.130:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 08:21:34.892922 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 18442
I20260619 08:21:34.898586 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:33625
--webserver_interface=127.4.151.190
--webserver_port=40673
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:33625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:21:34.986101 18977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:34.986403 18977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:34.986447 18977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:34.987749 18977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:21:34.987836 18977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:34.987862 18977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:34.987886 18977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:21:34.987913 18977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:21:34.989755 18977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:33625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:33625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=40673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:34.990053 18977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:34.990319 18977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:35.003001 18986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:35.003021 18984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:35.003001 18983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:35.003449 18977 server_base.cc:1034] running on GCE node
I20260619 08:21:35.003786 18977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:35.004153 18977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:35.005465 18977 hybrid_clock.cc:648] HybridClock initialized: now 1781857295005431 us; error 47 us; skew 500 ppm
Jun 19 08:21:35 dist-test-slave-d3n8 krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857295, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:35.008586 18977 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:21:35.009123 18977 webserver.cc:460] Webserver started at http://127.4.151.190:40673/ using document root <none> and password file <none>
I20260619 08:21:35.009510 18977 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:35.009577 18977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:21:35.011268 18977 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:21:35.012868 18993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:35.013295 18977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:21:35.013474 18977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "122c5c691eca4a27a136540a47e5261a"
format_stamp: "Formatted at 2026-06-19 08:21:33 on dist-test-slave-d3n8"
server_key: "d2e3ecd554c588c86a6d2683bdb1ce25"
server_key_iv: "337374d5b886b57363eaca9421c4b0a2"
server_key_version: "encryptionkey@0"
I20260619 08:21:35.014031 18977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:35.039393 18977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:35.042100 18977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:35.042285 18977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:35.046511 18977 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:33625
I20260619 08:21:35.046550 19045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:33625 every 8 connection(s)
I20260619 08:21:35.046965 18977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:21:35.048859 19046 sys_catalog.cc:263] Verifying existing consensus state
I20260619 08:21:35.049746 19046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a: Bootstrap starting.
I20260619 08:21:35.055369 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 18977
I20260619 08:21:35.055773 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:33811
--local_ip_for_outbound_sockets=127.4.151.129
--tserver_master_addrs=127.4.151.190:33625
--webserver_port=44795
--webserver_interface=127.4.151.129
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:21:35.059784 19046 log.cc:824] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a: Log is configured to *not* fsync() on all Append() calls
I20260619 08:21:35.062083 19046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:21:35.062474 19046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a: Bootstrap complete.
I20260619 08:21:35.065085 19046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122c5c691eca4a27a136540a47e5261a" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33625 } }
I20260619 08:21:35.065346 19046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122c5c691eca4a27a136540a47e5261a, State: Initialized, Role: FOLLOWER
I20260619 08:21:35.065649 19046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122c5c691eca4a27a136540a47e5261a" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33625 } }
I20260619 08:21:35.065739 19046 raft_consensus.cc:388] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:21:35.065778 19046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:21:35.065863 19046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [term 1 FOLLOWER]: Advancing to term 2
I20260619 08:21:35.067059 19046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122c5c691eca4a27a136540a47e5261a" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33625 } }
I20260619 08:21:35.067160 19046 leader_election.cc:302] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 122c5c691eca4a27a136540a47e5261a; no voters:
I20260619 08:21:35.067456 19046 leader_election.cc:288] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 08:21:35.067494 19050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [term 2 FOLLOWER]: Leader election won for term 2
I20260619 08:21:35.067610 19050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [term 2 LEADER]: Becoming Leader. State: Replica: 122c5c691eca4a27a136540a47e5261a, State: Running, Role: LEADER
I20260619 08:21:35.067767 19046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:21:35.067796 19050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122c5c691eca4a27a136540a47e5261a" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33625 } }
I20260619 08:21:35.068254 19051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "122c5c691eca4a27a136540a47e5261a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122c5c691eca4a27a136540a47e5261a" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33625 } } }
I20260619 08:21:35.068346 19051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [sys.catalog]: This master's current role is: LEADER
I20260619 08:21:35.068764 19052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 122c5c691eca4a27a136540a47e5261a. Latest consensus state: current_term: 2 leader_uuid: "122c5c691eca4a27a136540a47e5261a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122c5c691eca4a27a136540a47e5261a" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33625 } } }
I20260619 08:21:35.068845 19052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [sys.catalog]: This master's current role is: LEADER
I20260619 08:21:35.068987 19056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:21:35.069960 19056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:21:35.070123 19056 catalog_manager.cc:1259] Loaded cluster ID: 8a85f3056c48428ea220d6bcebaa75ca
I20260619 08:21:35.070364 19056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:21:35.070643 19056 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 08:21:35.076071 19056 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 08:21:35.078994 19056 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:21:35.079254 19056 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a: Loaded TSK: 0
I20260619 08:21:35.079514 19056 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 08:21:35.144639 19048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:35.144908 19048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:35.144950 19048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:35.146301 19048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:35.146381 19048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:35.146430 19048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:21:35.148052 19048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:33811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=44795
--tserver_master_addrs=127.4.151.190:33625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:35.148365 19048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:35.148617 19048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:35.154251 19074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:35.154259 19076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:35.154237 19073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:35.155635 19048 server_base.cc:1034] running on GCE node
I20260619 08:21:35.155869 19048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:35.156143 19048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:35.157325 19048 hybrid_clock.cc:648] HybridClock initialized: now 1781857295157309 us; error 33 us; skew 500 ppm
Jun 19 08:21:35 dist-test-slave-d3n8 krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857295, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:35.160050 19048 init.cc:377] Logged in from keytab as kudu/127.4.151.129@KRBTEST.COM (short username kudu)
I20260619 08:21:35.160581 19048 webserver.cc:460] Webserver started at http://127.4.151.129:44795/ using document root <none> and password file <none>
I20260619 08:21:35.160840 19048 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:35.160897 19048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:21:35.162396 19048 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:21:35.163143 19083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:35.163404 19048 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:21:35.163508 19048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "93af7b16aab148ec9d117f477a745284"
format_stamp: "Formatted at 2026-06-19 08:21:34 on dist-test-slave-d3n8"
server_key: "630de7b74e948fa46757e98c77d7bf23"
server_key_iv: "8819b7971ea9a1e5de7d1f546ea779fa"
server_key_version: "encryptionkey@0"
I20260619 08:21:35.163884 19048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:35.197314 19048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:35.200030 19048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:35.200233 19048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:35.200492 19048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:21:35.200961 19090 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 08:21:35.203157 19048 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 08:21:35.203253 19048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 08:21:35.203305 19048 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 08:21:35.203974 19048 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 08:21:35.204041 19048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 08:21:35.204101 19090 tablet_bootstrap.cc:492] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284: Bootstrap starting.
I20260619 08:21:35.211338 19048 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:33811
I20260619 08:21:35.211419 19197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:33811 every 8 connection(s)
I20260619 08:21:35.211760 19048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
I20260619 08:21:35.212235 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 19048
I20260619 08:21:35.212589 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:43707
--local_ip_for_outbound_sockets=127.4.151.130
--tserver_master_addrs=127.4.151.190:33625
--webserver_port=41893
--webserver_interface=127.4.151.130
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:21:35.214988 19090 log.cc:824] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284: Log is configured to *not* fsync() on all Append() calls
I20260619 08:21:35.217748 19090 tablet_bootstrap.cc:492] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:21:35.218091 19090 tablet_bootstrap.cc:492] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284: Bootstrap complete.
I20260619 08:21:35.219187 19090 ts_tablet_manager.cc:1397] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20260619 08:21:35.220709 19090 raft_consensus.cc:348] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
Jun 19 08:21:35 dist-test-slave-d3n8 krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857295, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:21:35.221047 19090 raft_consensus.cc:729] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93af7b16aab148ec9d117f477a745284, State: Initialized, Role: FOLLOWER
I20260619 08:21:35.221908 19090 consensus_queue.cc:260] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:35.222245 19090 ts_tablet_manager.cc:1428] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260619 08:21:35.226332 19200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:35.212593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.129:35703)
0619 08:21:35.213500 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:35.213560 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:35.214411 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:35.214949 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:35.214953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:35.215332 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:35.215604 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:35.215610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.217251 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.217252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:35.218609 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:35.218613 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.219578 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.219579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:35.219734 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:35.220388 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:35.220398 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:35.222150 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:35.224535 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:35.224542 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:35.224544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:35.224796 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:35.225106 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:35.225107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:35.225108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:35.225221 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:35.225781 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:35.225788 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:35.225908 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 08:21:35.226009 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 08:21:35.226147 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"thread_start_us":110,"threads_started":1}
I20260619 08:21:35.226432 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:35.212702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:35703 (local address 127.4.151.190:33625)
0619 08:21:35.213144 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:35.213157 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:35.213690 (+ 533us) server_negotiation.cc:407] Connection header received
0619 08:21:35.214680 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:35.214688 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:35.214753 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:35.214904 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:35.215734 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.217101 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.218738 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.219446 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.219588 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:35.222310 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:35.222316 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:35.222316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:35.222330 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:35.224427 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:35.224891 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:35.224895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:35.224895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:35.225005 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:35.225303 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:35.225303 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:35.225304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:35.225660 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:35.225746 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:35.226100 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 08:21:35.226241 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":94,"threads_started":1}
I20260619 08:21:35.227186 19198 heartbeater.cc:347] Connected to a master server at 127.4.151.190:33625
I20260619 08:21:35.227308 19198 heartbeater.cc:464] Registering TS with master...
I20260619 08:21:35.227483 19198 heartbeater.cc:511] Master 127.4.151.190:33625 requested a full tablet report, sending...
I20260619 08:21:35.228228 19010 ts_manager.cc:194] Registered new tserver with Master: 93af7b16aab148ec9d117f477a745284 (127.4.151.129:33811)
I20260619 08:21:35.229483 19010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.129@KRBTEST.COM'} at 127.4.151.129:35703
I20260619 08:21:35.230458 19198 heartbeater.cc:503] Master 127.4.151.190:33625 was elected leader, sending a full tablet report...
Jun 19 08:21:35 dist-test-slave-d3n8 krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857295, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for kudu/127.4.151.129@KRBTEST.COM
I20260619 08:21:35.237429 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:35.228872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:33811 (local address 127.0.0.1:33894)
0619 08:21:35.229178 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:35.229204 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:35.229297 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:35.229529 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:35.229534 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:35.229886 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:35.229953 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:35.229955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.230989 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.230990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:35.231563 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:35.231565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.232292 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.232293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:35.232350 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:35.232889 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:35.232893 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:35.234212 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:35.236206 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:35.236213 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:35.236215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:35.236418 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:35.236653 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:35.236654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:35.236654 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:35.236709 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:35.237088 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:35.237093 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:35.237138 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 08:21:35.237203 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 08:21:35.237306 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":106,"threads_started":1}
I20260619 08:21:35.237488 19206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:35.228927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33894 (local address 127.4.151.129:33811)
0619 08:21:35.229127 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:35.229131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:35.229273 (+ 142us) server_negotiation.cc:407] Connection header received
0619 08:21:35.229423 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:35.229426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:35.229442 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:35.229528 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:35.230045 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.230903 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.231650 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.232120 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.232232 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:35.234301 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:35.234308 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:35.234309 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:35.234316 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:35.236112 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:35.236502 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:35.236505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:35.236505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:35.236583 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:35.236830 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:35.236831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:35.236831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:35.237016 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:35.237109 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:35.237265 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 08:21:35.237364 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":173,"thread_start_us":67,"threads_started":1}
I20260619 08:21:35.238299 19132 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fc92b22dabd430abc58ddcfebff2ed7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:21:34 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:33894
I20260619 08:21:35.238941 19207 tablet_replica.cc:330] stopping tablet replica
I20260619 08:21:35.239156 19207 raft_consensus.cc:2229] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:21:35.239250 19207 raft_consensus.cc:2258] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:21:35.239563 19207 ts_tablet_manager.cc:1905] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:21:35.241905 19207 ts_tablet_manager.cc:1918] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:21:35.241974 19207 log.cc:1196] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/wal/wals/3fc92b22dabd430abc58ddcfebff2ed7
I20260619 08:21:35.242202 19207 ts_tablet_manager.cc:1939] T 3fc92b22dabd430abc58ddcfebff2ed7 P 93af7b16aab148ec9d117f477a745284: Deleting consensus metadata
I20260619 08:21:35.242722 18996 catalog_manager.cc:4841] TS 93af7b16aab148ec9d117f477a745284 (127.4.151.129:33811): tablet 3fc92b22dabd430abc58ddcfebff2ed7 (table test-table [id=b783aa3dcdfe4a9ca78980eafd76d392]) successfully deleted
W20260619 08:21:35.303368 19202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:35.303653 19202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:35.303697 19202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:35.304951 19202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:35.305037 19202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:35.305085 19202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:21:35.307035 19202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:43707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=41893
--tserver_master_addrs=127.4.151.190:33625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:35.307348 19202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:35.307587 19202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:35.313848 19212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:35.313840 19213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:35.313886 19202 server_base.cc:1034] running on GCE node
W20260619 08:21:35.313859 19215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:35.314297 19202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:35.314594 19202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:35.315795 19202 hybrid_clock.cc:648] HybridClock initialized: now 1781857295315779 us; error 64 us; skew 500 ppm
Jun 19 08:21:35 dist-test-slave-d3n8 krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857295, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:35.318660 19202 init.cc:377] Logged in from keytab as kudu/127.4.151.130@KRBTEST.COM (short username kudu)
I20260619 08:21:35.319134 19202 webserver.cc:460] Webserver started at http://127.4.151.130:41893/ using document root <none> and password file <none>
I20260619 08:21:35.319384 19202 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:35.319468 19202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:21:35.320900 19202 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:21:35.321771 19222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:35.321959 19202 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:21:35.322057 19202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "8b2473c351fe4d4a9be83937dbbc9f5b"
format_stamp: "Formatted at 2026-06-19 08:21:34 on dist-test-slave-d3n8"
server_key: "c83a18e6fa9c6720886437006150bddb"
server_key_iv: "70fd3c6974b6d49d2c471c88d85854e3"
server_key_version: "encryptionkey@0"
I20260619 08:21:35.322440 19202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:35.350888 19202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:35.353456 19202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:35.353667 19202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:35.353945 19202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:21:35.354466 19229 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 08:21:35.356096 19202 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 08:21:35.356194 19202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 08:21:35.356249 19202 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 08:21:35.356931 19202 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 08:21:35.356994 19202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 08:21:35.357076 19229 tablet_bootstrap.cc:492] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b: Bootstrap starting.
I20260619 08:21:35.364059 19202 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:43707
I20260619 08:21:35.364096 19336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:43707 every 8 connection(s)
I20260619 08:21:35.364456 19202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
I20260619 08:21:35.368594 19229 log.cc:824] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b: Log is configured to *not* fsync() on all Append() calls
I20260619 08:21:35.369072 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 19202
I20260619 08:21:35.369501 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:33519
--local_ip_for_outbound_sockets=127.4.151.131
--tserver_master_addrs=127.4.151.190:33625
--webserver_port=45601
--webserver_interface=127.4.151.131
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:21:35.371544 19229 tablet_bootstrap.cc:492] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:21:35.371901 19229 tablet_bootstrap.cc:492] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b: Bootstrap complete.
Jun 19 08:21:35 dist-test-slave-d3n8 krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857295, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:21:35.373139 19229 ts_tablet_manager.cc:1397] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.003s
I20260619 08:21:35.374946 19229 raft_consensus.cc:348] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:35.375418 19229 raft_consensus.cc:729] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b2473c351fe4d4a9be83937dbbc9f5b, State: Initialized, Role: FOLLOWER
I20260619 08:21:35.375638 19229 consensus_queue.cc:260] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:35.375972 19229 ts_tablet_manager.cc:1428] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20260619 08:21:35.376579 19339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:35.365235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.130:42563)
0619 08:21:35.365437 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:35.365480 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:35.366301 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:35.366655 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:35.366660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:35.367156 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:35.367461 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:35.367472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.368339 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.368340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:35.369603 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:35.369606 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.370230 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.370231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:35.370363 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:35.370990 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:35.371000 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:35.372907 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:35.374779 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:35.374785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:35.374788 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:35.375152 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:35.375374 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:35.375375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:35.375376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:35.375511 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:35.375832 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:35.375838 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:35.376088 (+ 250us) client_negotiation.cc:769] Sending connection context
0619 08:21:35.376280 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 08:21:35.376429 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":104,"threads_started":1}
I20260619 08:21:35.376610 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:35.365284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:42563 (local address 127.4.151.190:33625)
0619 08:21:35.365356 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:35.365358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:35.365575 (+ 217us) server_negotiation.cc:407] Connection header received
0619 08:21:35.366470 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:35.366471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:35.366487 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:35.366550 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:35.367581 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.368236 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.369715 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.370042 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.370108 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:35.373021 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:35.373023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:35.373023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:35.373031 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:35.374660 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:35.375241 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:35.375243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:35.375243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:35.375304 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:35.375572 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:35.375574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:35.375574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:35.375712 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:35.375804 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:35.376347 (+ 543us) server_negotiation.cc:299] Negotiation successful
0619 08:21:35.376461 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 08:21:35.377151 19337 heartbeater.cc:347] Connected to a master server at 127.4.151.190:33625
I20260619 08:21:35.377262 19337 heartbeater.cc:464] Registering TS with master...
I20260619 08:21:35.377485 19337 heartbeater.cc:511] Master 127.4.151.190:33625 requested a full tablet report, sending...
I20260619 08:21:35.378032 19010 ts_manager.cc:194] Registered new tserver with Master: 8b2473c351fe4d4a9be83937dbbc9f5b (127.4.151.130:43707)
I20260619 08:21:35.378707 19010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.130@KRBTEST.COM'} at 127.4.151.130:42563
I20260619 08:21:35.380115 19337 heartbeater.cc:503] Master 127.4.151.190:33625 was elected leader, sending a full tablet report...
Jun 19 08:21:35 dist-test-slave-d3n8 krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857295, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for kudu/127.4.151.130@KRBTEST.COM
I20260619 08:21:35.386152 19343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:35.378526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50778 (local address 127.4.151.130:43707)
0619 08:21:35.378686 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:35.378690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:35.378699 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:21:35.378732 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:35.378738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:35.378752 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:35.378869 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:35.379180 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.379910 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.380628 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.380937 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.381028 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:35.383189 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:35.383197 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:35.383197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:35.383205 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:35.384776 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:35.385194 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:35.385197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:35.385197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:35.385254 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:35.385512 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:35.385513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:35.385513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:35.385677 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:35.385773 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:35.385956 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 08:21:35.386050 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":104,"threads_started":1}
I20260619 08:21:35.386145 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:35.378355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:43707 (local address 127.0.0.1:50778)
0619 08:21:35.378410 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:35.378421 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:35.378464 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:35.378878 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:35.378879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:35.379036 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:35.379084 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:35.379086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.379998 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.379999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:35.380536 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:35.380538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.381018 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.381019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:35.381086 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:35.381639 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:35.381644 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:35.383084 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:35.384887 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:35.384888 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:35.384888 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:35.385102 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:35.385325 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:35.385326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:35.385326 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:35.385445 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:35.385753 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:35.385754 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:35.385801 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 08:21:35.385915 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 08:21:35.386033 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 08:21:35.386562 19271 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fc92b22dabd430abc58ddcfebff2ed7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:21:34 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:50778
I20260619 08:21:35.387101 19344 tablet_replica.cc:330] stopping tablet replica
I20260619 08:21:35.387230 19344 raft_consensus.cc:2229] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:21:35.387307 19344 raft_consensus.cc:2258] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:21:35.387643 19344 ts_tablet_manager.cc:1905] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:21:35.390363 19344 ts_tablet_manager.cc:1918] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:21:35.390450 19344 log.cc:1196] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/3fc92b22dabd430abc58ddcfebff2ed7
I20260619 08:21:35.390676 19344 ts_tablet_manager.cc:1939] T 3fc92b22dabd430abc58ddcfebff2ed7 P 8b2473c351fe4d4a9be83937dbbc9f5b: Deleting consensus metadata
I20260619 08:21:35.391283 18995 catalog_manager.cc:4841] TS 8b2473c351fe4d4a9be83937dbbc9f5b (127.4.151.130:43707): tablet 3fc92b22dabd430abc58ddcfebff2ed7 (table test-table [id=b783aa3dcdfe4a9ca78980eafd76d392]) successfully deleted
W20260619 08:21:35.460366 19341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:35.460595 19341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:35.460621 19341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:35.461900 19341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:35.461951 19341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:35.461983 19341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:21:35.463503 19341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:33519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=45601
--tserver_master_addrs=127.4.151.190:33625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:35.463712 19341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:35.463910 19341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:35.469252 19352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:35.469292 19349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:35.469300 19350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:35.469789 19341 server_base.cc:1034] running on GCE node
I20260619 08:21:35.470005 19341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:35.470285 19341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:35.471468 19341 hybrid_clock.cc:648] HybridClock initialized: now 1781857295471471 us; error 48 us; skew 500 ppm
Jun 19 08:21:35 dist-test-slave-d3n8 krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857295, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:35.474423 19341 init.cc:377] Logged in from keytab as kudu/127.4.151.131@KRBTEST.COM (short username kudu)
I20260619 08:21:35.474931 19341 webserver.cc:460] Webserver started at http://127.4.151.131:45601/ using document root <none> and password file <none>
I20260619 08:21:35.475214 19341 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:35.475304 19341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:21:35.477536 19341 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:21:35.478294 19359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:35.478488 19341 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:21:35.478608 19341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "d315c1374d844a9ab7774177f8150672"
format_stamp: "Formatted at 2026-06-19 08:21:34 on dist-test-slave-d3n8"
server_key: "793bdcbcef4d0dca95a2c22a17abc9c0"
server_key_iv: "ef676be221720c2b6e670f1e463e992c"
server_key_version: "encryptionkey@0"
I20260619 08:21:35.479012 19341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:35.487696 19341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:35.490221 19341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:35.490386 19341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:35.490633 19341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:21:35.490955 19341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:21:35.491011 19341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:35.491060 19341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:21:35.491089 19341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:35.497022 19341 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:33519
I20260619 08:21:35.497061 19472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:33519 every 8 connection(s)
I20260619 08:21:35.497498 19341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
Jun 19 08:21:35 dist-test-slave-d3n8 krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857295, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:21:35.507195 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 19341
I20260619 08:21:35.508064 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:35.498385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:39501 (local address 127.4.151.190:33625)
0619 08:21:35.498474 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:35.498477 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:35.498713 (+ 236us) server_negotiation.cc:407] Connection header received
0619 08:21:35.499317 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:35.499318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:35.499328 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:35.499415 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:35.500535 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.501099 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.502283 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.502509 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.502621 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:35.504922 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:35.504923 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:35.504924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:35.504930 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:35.506450 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:35.506851 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:35.506852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:35.506852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:35.506909 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:35.507164 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:35.507165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:35.507166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:35.507300 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:35.507384 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:35.507835 (+ 451us) server_negotiation.cc:299] Negotiation successful
0619 08:21:35.507939 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 08:21:35.508260 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:35.498368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.131:39501)
0619 08:21:35.498570 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:35.498618 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:35.499137 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:35.499516 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:35.499521 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:35.500091 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:35.500297 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:35.500301 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.501189 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.501240 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:35.502166 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:35.502169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.502629 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.502630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:35.502766 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:35.503317 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:35.503325 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:35.504822 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:35.506533 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:35.506539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:35.506541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:35.506775 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:35.506975 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:35.506975 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:35.506976 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:35.507074 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:35.507412 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:35.507420 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:35.507609 (+ 189us) client_negotiation.cc:769] Sending connection context
0619 08:21:35.507777 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 08:21:35.507985 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":123,"threads_started":1}
I20260619 08:21:35.508708 19473 heartbeater.cc:347] Connected to a master server at 127.4.151.190:33625
I20260619 08:21:35.508792 19473 heartbeater.cc:464] Registering TS with master...
I20260619 08:21:35.509021 19473 heartbeater.cc:511] Master 127.4.151.190:33625 requested a full tablet report, sending...
I20260619 08:21:35.509500 19010 ts_manager.cc:194] Registered new tserver with Master: d315c1374d844a9ab7774177f8150672 (127.4.151.131:33519)
I20260619 08:21:35.509902 19010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.131@KRBTEST.COM'} at 127.4.151.131:39501
I20260619 08:21:35.515131 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:35.507725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46460 (local address 127.4.151.190:33625)
0619 08:21:35.507851 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:35.507853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:35.507990 (+ 137us) server_negotiation.cc:407] Connection header received
0619 08:21:35.508057 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:35.508058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:35.508071 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:35.508136 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:35.509259 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.509908 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.510823 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.511024 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.511080 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:35.512244 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:35.512246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:35.512246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:35.512253 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:35.513836 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:35.514298 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:35.514299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:35.514300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:35.514355 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:35.514612 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:35.514613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:35.514613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:35.514737 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:35.514864 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:35.514982 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 08:21:35.515059 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
I20260619 08:21:35.516057 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:21:35.519408 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:35.516533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46466 (local address 127.4.151.190:33625)
0619 08:21:35.516665 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:35.516666 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:35.516745 (+ 79us) server_negotiation.cc:407] Connection header received
0619 08:21:35.516780 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:35.516781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:35.516857 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:35.516943 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:21:35.517121 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.517748 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.518568 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.518781 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.518848 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:35.518996 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:21:35.519115 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:21:35.519183 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:35.519328 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 08:21:35.519352 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260619 08:21:35.519785 19010 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:46466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:21:35.521430 19010 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:21:35.523447 19132 tablet_service.cc:1467] Processing CreateTablet for tablet 96ef5d62f07d4675a10a88cf6fd2029e (DEFAULT_TABLE table=test-table [id=d7ccd70134d54ca2878a5146251b999c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:21:35.523597 19271 tablet_service.cc:1467] Processing CreateTablet for tablet 96ef5d62f07d4675a10a88cf6fd2029e (DEFAULT_TABLE table=test-table [id=d7ccd70134d54ca2878a5146251b999c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:21:35.523653 19132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ef5d62f07d4675a10a88cf6fd2029e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:21:35.523777 19271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ef5d62f07d4675a10a88cf6fd2029e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:21:35.525224 19229 tablet_bootstrap.cc:492] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b: Bootstrap starting.
I20260619 08:21:35.525297 19090 tablet_bootstrap.cc:492] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284: Bootstrap starting.
I20260619 08:21:35.525568 19229 tablet_bootstrap.cc:654] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b: Neither blocks nor log segments found. Creating new log.
I20260619 08:21:35.525624 19090 tablet_bootstrap.cc:654] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284: Neither blocks nor log segments found. Creating new log.
I20260619 08:21:35.526363 19090 tablet_bootstrap.cc:492] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284: No bootstrap required, opened a new log
I20260619 08:21:35.526418 19090 ts_tablet_manager.cc:1397] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:21:35.526782 19090 raft_consensus.cc:348] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:35.526847 19090 raft_consensus.cc:374] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:21:35.526880 19090 raft_consensus.cc:729] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93af7b16aab148ec9d117f477a745284, State: Initialized, Role: FOLLOWER
I20260619 08:21:35.526942 19090 consensus_queue.cc:260] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:35.527029 19090 ts_tablet_manager.cc:1428] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 08:21:35.527294 19229 tablet_bootstrap.cc:492] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b: No bootstrap required, opened a new log
I20260619 08:21:35.527349 19229 ts_tablet_manager.cc:1397] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:21:35.527523 19229 raft_consensus.cc:348] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:35.527603 19229 raft_consensus.cc:374] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:21:35.527678 19229 raft_consensus.cc:729] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b2473c351fe4d4a9be83937dbbc9f5b, State: Initialized, Role: FOLLOWER
I20260619 08:21:35.527741 19229 consensus_queue.cc:260] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:35.527776 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:35.523287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:33519 (local address 127.0.0.1:59602)
0619 08:21:35.523325 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:35.523337 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:35.523377 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:35.523797 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:35.523798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:35.523799 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:21:35.523843 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:35.523844 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.525269 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.525270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:35.526489 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:35.526493 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.527485 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.527486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:35.527558 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:35.527578 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 08:21:35.527648 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 08:21:35.527696 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260619 08:21:35.527835 19229 ts_tablet_manager.cc:1428] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:35.528029 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:35.523391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59602 (local address 127.4.151.131:33519)
0619 08:21:35.523595 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:35.523600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:35.523611 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:21:35.523651 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:35.523653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:35.523670 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:35.523769 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:21:35.523934 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.525141 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.526635 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.527385 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.527456 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:35.527528 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:35.527876 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 08:21:35.527929 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":104,"threads_started":1}
I20260619 08:21:35.528755 19407 tablet_service.cc:1467] Processing CreateTablet for tablet 96ef5d62f07d4675a10a88cf6fd2029e (DEFAULT_TABLE table=test-table [id=d7ccd70134d54ca2878a5146251b999c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:21:35.529083 19407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ef5d62f07d4675a10a88cf6fd2029e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:21:35.531905 19482 tablet_bootstrap.cc:492] T 96ef5d62f07d4675a10a88cf6fd2029e P d315c1374d844a9ab7774177f8150672: Bootstrap starting.
I20260619 08:21:35.532644 19482 tablet_bootstrap.cc:654] T 96ef5d62f07d4675a10a88cf6fd2029e P d315c1374d844a9ab7774177f8150672: Neither blocks nor log segments found. Creating new log.
I20260619 08:21:35.533128 19482 log.cc:824] T 96ef5d62f07d4675a10a88cf6fd2029e P d315c1374d844a9ab7774177f8150672: Log is configured to *not* fsync() on all Append() calls
I20260619 08:21:35.533874 19482 tablet_bootstrap.cc:492] T 96ef5d62f07d4675a10a88cf6fd2029e P d315c1374d844a9ab7774177f8150672: No bootstrap required, opened a new log
I20260619 08:21:35.533931 19482 ts_tablet_manager.cc:1397] T 96ef5d62f07d4675a10a88cf6fd2029e P d315c1374d844a9ab7774177f8150672: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:21:35.535234 19482 raft_consensus.cc:348] T 96ef5d62f07d4675a10a88cf6fd2029e P d315c1374d844a9ab7774177f8150672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:35.535373 19482 raft_consensus.cc:374] T 96ef5d62f07d4675a10a88cf6fd2029e P d315c1374d844a9ab7774177f8150672 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:21:35.535396 19482 raft_consensus.cc:729] T 96ef5d62f07d4675a10a88cf6fd2029e P d315c1374d844a9ab7774177f8150672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d315c1374d844a9ab7774177f8150672, State: Initialized, Role: FOLLOWER
I20260619 08:21:35.535521 19482 consensus_queue.cc:260] T 96ef5d62f07d4675a10a88cf6fd2029e P d315c1374d844a9ab7774177f8150672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:35.535816 19482 ts_tablet_manager.cc:1428] T 96ef5d62f07d4675a10a88cf6fd2029e P d315c1374d844a9ab7774177f8150672: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:21:35.535840 19473 heartbeater.cc:503] Master 127.4.151.190:33625 was elected leader, sending a full tablet report...
W20260619 08:21:35.615316 19338 tablet.cc:2369] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:21:35.712555 19199 tablet.cc:2369] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:21:35.748121 19474 tablet.cc:2369] T 96ef5d62f07d4675a10a88cf6fd2029e P d315c1374d844a9ab7774177f8150672: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:21:35.826725 19342 raft_consensus.cc:482] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:21:35.826871 19342 raft_consensus.cc:504] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:35.827327 19342 leader_election.cc:288] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d315c1374d844a9ab7774177f8150672 (127.4.151.131:33519), 93af7b16aab148ec9d117f477a745284 (127.4.151.129:33811)
I20260619 08:21:35.830894 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:35.827646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:49305 (local address 127.4.151.129:33811)
0619 08:21:35.827787 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:35.827789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:35.827797 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:21:35.827832 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:35.827832 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:35.827852 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:35.827946 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:21:35.828121 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.828716 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.830061 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.830512 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.830630 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:35.830715 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:35.830770 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 08:21:35.830817 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260619 08:21:35.830893 19339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:35.827565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:33811 (local address 127.4.151.130:49305)
0619 08:21:35.827621 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:35.827633 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:35.827700 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:35.827954 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:35.827955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:35.827957 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:21:35.828041 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:35.828043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.828851 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.828852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:35.829952 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:35.829955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.830625 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.830626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:35.830696 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:35.830699 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:21:35.830750 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 08:21:35.830796 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":169}
I20260619 08:21:35.830899 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:35.827727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:33631 (local address 127.4.151.131:33519)
0619 08:21:35.827775 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:35.827777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:35.827872 (+ 95us) server_negotiation.cc:407] Connection header received
0619 08:21:35.827962 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:35.827963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:35.827972 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:35.828044 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:21:35.828300 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.828789 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.830071 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.830464 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.830579 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:35.830590 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:35.830775 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 08:21:35.830822 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 08:21:35.831040 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:35.827567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:33519 (local address 127.4.151.130:33631)
0619 08:21:35.827775 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:35.827785 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:35.827888 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:35.828062 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:35.828062 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:35.828064 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:21:35.828145 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:35.828218 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.828951 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.828952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:35.829947 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:35.829950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.830598 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.830599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:35.830675 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:35.830692 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 08:21:35.830768 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 08:21:35.830814 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":65,"thread_start_us":68,"threads_started":1}
I20260619 08:21:35.831408 19427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ef5d62f07d4675a10a88cf6fd2029e" candidate_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d315c1374d844a9ab7774177f8150672" is_pre_election: true
I20260619 08:21:35.831423 19152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ef5d62f07d4675a10a88cf6fd2029e" candidate_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93af7b16aab148ec9d117f477a745284" is_pre_election: true
I20260619 08:21:35.831547 19427 raft_consensus.cc:2454] T 96ef5d62f07d4675a10a88cf6fd2029e P d315c1374d844a9ab7774177f8150672 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b2473c351fe4d4a9be83937dbbc9f5b in term 0.
I20260619 08:21:35.831580 19152 raft_consensus.cc:2454] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b2473c351fe4d4a9be83937dbbc9f5b in term 0.
I20260619 08:21:35.831817 19225 leader_election.cc:302] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b2473c351fe4d4a9be83937dbbc9f5b, 93af7b16aab148ec9d117f477a745284; no voters:
I20260619 08:21:35.831955 19342 raft_consensus.cc:2781] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:21:35.832011 19342 raft_consensus.cc:482] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:21:35.832036 19342 raft_consensus.cc:3037] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:21:35.832772 19342 raft_consensus.cc:504] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:35.832924 19342 leader_election.cc:288] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b [CANDIDATE]: Term 1 election: Requested vote from peers d315c1374d844a9ab7774177f8150672 (127.4.151.131:33519), 93af7b16aab148ec9d117f477a745284 (127.4.151.129:33811)
I20260619 08:21:35.833115 19427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ef5d62f07d4675a10a88cf6fd2029e" candidate_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d315c1374d844a9ab7774177f8150672"
I20260619 08:21:35.833124 19152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ef5d62f07d4675a10a88cf6fd2029e" candidate_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93af7b16aab148ec9d117f477a745284"
I20260619 08:21:35.833199 19152 raft_consensus.cc:3037] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:21:35.833205 19427 raft_consensus.cc:3037] T 96ef5d62f07d4675a10a88cf6fd2029e P d315c1374d844a9ab7774177f8150672 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:21:35.833956 19152 raft_consensus.cc:2454] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b2473c351fe4d4a9be83937dbbc9f5b in term 1.
I20260619 08:21:35.834147 19427 raft_consensus.cc:2454] T 96ef5d62f07d4675a10a88cf6fd2029e P d315c1374d844a9ab7774177f8150672 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b2473c351fe4d4a9be83937dbbc9f5b in term 1.
I20260619 08:21:35.834163 19225 leader_election.cc:302] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b2473c351fe4d4a9be83937dbbc9f5b, 93af7b16aab148ec9d117f477a745284; no voters:
I20260619 08:21:35.834266 19342 raft_consensus.cc:2781] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:21:35.834442 19342 raft_consensus.cc:686] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b [term 1 LEADER]: Becoming Leader. State: Replica: 8b2473c351fe4d4a9be83937dbbc9f5b, State: Running, Role: LEADER
I20260619 08:21:35.834528 19342 consensus_queue.cc:237] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:35.835381 19010 catalog_manager.cc:5495] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b2473c351fe4d4a9be83937dbbc9f5b (127.4.151.130). New cstate: current_term: 1 leader_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } health_report { overall_health: HEALTHY } } }
I20260619 08:21:35.884286 19343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:35.881614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50794 (local address 127.4.151.130:43707)
0619 08:21:35.881657 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:35.881659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:35.881674 (+ 15us) server_negotiation.cc:407] Connection header received
0619 08:21:35.881709 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:35.881710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:35.881728 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:35.881790 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:21:35.881957 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.882565 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.883426 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:35.883690 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:35.883755 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:35.883906 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:21:35.884027 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:21:35.884086 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:35.884204 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 08:21:35.884224 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 08:21:35.886149 19152 raft_consensus.cc:1261] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Refusing update from remote peer 8b2473c351fe4d4a9be83937dbbc9f5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:21:35.886152 19427 raft_consensus.cc:1261] T 96ef5d62f07d4675a10a88cf6fd2029e P d315c1374d844a9ab7774177f8150672 [term 1 FOLLOWER]: Refusing update from remote peer 8b2473c351fe4d4a9be83937dbbc9f5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:21:35.886510 19342 consensus_queue.cc:1035] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:21:35.886658 19487 consensus_queue.cc:1035] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:21:35.891242 19009 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:46466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:21:35.891342 19009 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:46466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:21:35.892257 19009 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a: Sending DeleteTablet for 3 replicas of tablet 96ef5d62f07d4675a10a88cf6fd2029e
I20260619 08:21:35.892573 19132 tablet_service.cc:1514] Processing DeleteTablet for tablet 96ef5d62f07d4675a10a88cf6fd2029e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:21:35 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:33894
I20260619 08:21:35.892632 19271 tablet_service.cc:1514] Processing DeleteTablet for tablet 96ef5d62f07d4675a10a88cf6fd2029e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:21:35 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:50778
I20260619 08:21:35.892751 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 19048
I20260619 08:21:35.892800 19407 tablet_service.cc:1514] Processing DeleteTablet for tablet 96ef5d62f07d4675a10a88cf6fd2029e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:21:35 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:59602
I20260619 08:21:35.893419 19500 tablet_replica.cc:330] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b: stopping tablet replica
I20260619 08:21:35.893493 19500 raft_consensus.cc:2229] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:21:35.893610 19500 raft_consensus.cc:2258] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:21:35.894019 19500 ts_tablet_manager.cc:1905] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:21:35.896230 19500 ts_tablet_manager.cc:1918] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:21:35.896306 19500 log.cc:1196] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/96ef5d62f07d4675a10a88cf6fd2029e
I20260619 08:21:35.896564 19500 ts_tablet_manager.cc:1939] T 96ef5d62f07d4675a10a88cf6fd2029e P 8b2473c351fe4d4a9be83937dbbc9f5b: Deleting consensus metadata
I20260619 08:21:35.897081 18995 catalog_manager.cc:4841] TS 8b2473c351fe4d4a9be83937dbbc9f5b (127.4.151.130:43707): tablet 96ef5d62f07d4675a10a88cf6fd2029e (table test-table [id=d7ccd70134d54ca2878a5146251b999c]) successfully deleted
W20260619 08:21:35.900219 18996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.151.129:33811 (error 108)
I20260619 08:21:35.900345 19501 tablet_replica.cc:330] T 96ef5d62f07d4675a10a88cf6fd2029e P d315c1374d844a9ab7774177f8150672: stopping tablet replica
I20260619 08:21:35.900486 19501 raft_consensus.cc:2229] T 96ef5d62f07d4675a10a88cf6fd2029e P d315c1374d844a9ab7774177f8150672 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:21:35.900636 19501 raft_consensus.cc:2258] T 96ef5d62f07d4675a10a88cf6fd2029e P d315c1374d844a9ab7774177f8150672 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:21:35.900637 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 19202
I20260619 08:21:35.901237 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:35.900810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:33811 (local address 127.0.0.1:33910)
0619 08:21:35.901158 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:35.901203 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.129:33811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327}
W20260619 08:21:35.901481 18996 catalog_manager.cc:4584] TS 93af7b16aab148ec9d117f477a745284 (127.4.151.129:33811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 96ef5d62f07d4675a10a88cf6fd2029e: Network error: Client connection negotiation failed: client connection to 127.4.151.129:33811: connect: Connection refused (error 111)
I20260619 08:21:35.902559 19501 ts_tablet_manager.cc:1905] T 96ef5d62f07d4675a10a88cf6fd2029e P d315c1374d844a9ab7774177f8150672: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:21:35.904253 19501 ts_tablet_manager.cc:1918] T 96ef5d62f07d4675a10a88cf6fd2029e P d315c1374d844a9ab7774177f8150672: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:21:35.904379 19501 log.cc:1196] T 96ef5d62f07d4675a10a88cf6fd2029e P d315c1374d844a9ab7774177f8150672: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/96ef5d62f07d4675a10a88cf6fd2029e
I20260619 08:21:35.904647 19501 ts_tablet_manager.cc:1939] T 96ef5d62f07d4675a10a88cf6fd2029e P d315c1374d844a9ab7774177f8150672: Deleting consensus metadata
I20260619 08:21:35.905140 18995 catalog_manager.cc:4841] TS d315c1374d844a9ab7774177f8150672 (127.4.151.131:33519): tablet 96ef5d62f07d4675a10a88cf6fd2029e (table test-table [id=d7ccd70134d54ca2878a5146251b999c]) successfully deleted
I20260619 08:21:35.905977 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 19341
I20260619 08:21:35.910360 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 18977
I20260619 08:21:35.915728 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:33625
--webserver_interface=127.4.151.190
--webserver_port=40673
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:33625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:21:35.999433 19503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:35.999670 19503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:35.999701 19503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:36.000826 19503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:21:36.000861 19503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:36.000875 19503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:36.000888 19503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:21:36.000900 19503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:21:36.002345 19503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:33625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:33625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=40673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:36.002550 19503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:36.002730 19503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:36.007968 19511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:36.007985 19508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:36.008092 19509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:36.008251 19503 server_base.cc:1034] running on GCE node
I20260619 08:21:36.008502 19503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:36.008836 19503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:36.010005 19503 hybrid_clock.cc:648] HybridClock initialized: now 1781857296009990 us; error 36 us; skew 500 ppm
Jun 19 08:21:36 dist-test-slave-d3n8 krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857296, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:36.012933 19503 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:21:36.013499 19503 webserver.cc:460] Webserver started at http://127.4.151.190:40673/ using document root <none> and password file <none>
I20260619 08:21:36.013777 19503 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:36.013827 19503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:21:36.015239 19503 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:21:36.015964 19518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:36.016202 19503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:21:36.016311 19503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "122c5c691eca4a27a136540a47e5261a"
format_stamp: "Formatted at 2026-06-19 08:21:33 on dist-test-slave-d3n8"
server_key: "d2e3ecd554c588c86a6d2683bdb1ce25"
server_key_iv: "337374d5b886b57363eaca9421c4b0a2"
server_key_version: "encryptionkey@0"
I20260619 08:21:36.016726 19503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:36.025873 19503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:36.027901 19503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:36.028192 19503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:36.032878 19503 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:33625
I20260619 08:21:36.032909 19570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:33625 every 8 connection(s)
I20260619 08:21:36.033320 19503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:21:36.035051 19571 sys_catalog.cc:263] Verifying existing consensus state
I20260619 08:21:36.035729 19571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a: Bootstrap starting.
I20260619 08:21:36.041283 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 19503
I20260619 08:21:36.041639 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:33811
--local_ip_for_outbound_sockets=127.4.151.129
--tserver_master_addrs=127.4.151.190:33625
--webserver_port=44795
--webserver_interface=127.4.151.129
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:21:36.044387 19571 log.cc:824] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a: Log is configured to *not* fsync() on all Append() calls
I20260619 08:21:36.046911 19571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:21:36.047245 19571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a: Bootstrap complete.
I20260619 08:21:36.049640 19571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122c5c691eca4a27a136540a47e5261a" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33625 } }
I20260619 08:21:36.049875 19571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122c5c691eca4a27a136540a47e5261a, State: Initialized, Role: FOLLOWER
I20260619 08:21:36.050043 19571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122c5c691eca4a27a136540a47e5261a" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33625 } }
I20260619 08:21:36.050112 19571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:21:36.050173 19571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:21:36.050225 19571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [term 2 FOLLOWER]: Advancing to term 3
I20260619 08:21:36.051290 19571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122c5c691eca4a27a136540a47e5261a" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33625 } }
I20260619 08:21:36.051379 19571 leader_election.cc:302] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 122c5c691eca4a27a136540a47e5261a; no voters:
I20260619 08:21:36.051594 19571 leader_election.cc:288] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 08:21:36.051635 19575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [term 3 FOLLOWER]: Leader election won for term 3
I20260619 08:21:36.051750 19575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [term 3 LEADER]: Becoming Leader. State: Replica: 122c5c691eca4a27a136540a47e5261a, State: Running, Role: LEADER
I20260619 08:21:36.051894 19571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:21:36.051909 19575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122c5c691eca4a27a136540a47e5261a" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33625 } }
I20260619 08:21:36.052331 19577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 122c5c691eca4a27a136540a47e5261a. Latest consensus state: current_term: 3 leader_uuid: "122c5c691eca4a27a136540a47e5261a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122c5c691eca4a27a136540a47e5261a" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33625 } } }
I20260619 08:21:36.052419 19577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [sys.catalog]: This master's current role is: LEADER
I20260619 08:21:36.052739 19576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "122c5c691eca4a27a136540a47e5261a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122c5c691eca4a27a136540a47e5261a" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33625 } } }
I20260619 08:21:36.052804 19576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 122c5c691eca4a27a136540a47e5261a [sys.catalog]: This master's current role is: LEADER
I20260619 08:21:36.052940 19582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:21:36.053817 19582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:21:36.053964 19582 catalog_manager.cc:1259] Loaded cluster ID: 8a85f3056c48428ea220d6bcebaa75ca
I20260619 08:21:36.054198 19582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 08:21:36.054441 19582 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f25292bb9ad google::LogMessage::Fail() at ??:0
@ 0x7f25292be88a google::LogMessage::SendToLog() at ??:0
@ 0x7f25292bb4de google::LogMessage::Flush() at ??:0
@ 0x7f25292bca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f25311dc0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f25311e58b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f252b1890d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f252b17e954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f252bfe26db start_thread at ??:0
@ 0x7f2527db961f clone at ??:0
W20260619 08:21:36.156042 19573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:36.156354 19573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:36.156394 19573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:36.158231 19573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:36.158304 19573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:36.158349 19573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:21:36.160604 19573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:33811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=44795
--tserver_master_addrs=127.4.151.190:33625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:36.160882 19573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:36.161180 19573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:36.166781 19610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:36.166792 19607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:36.166980 19573 server_base.cc:1034] running on GCE node
W20260619 08:21:36.167114 19608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:36.167372 19573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:36.167634 19573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:36.168852 19573 hybrid_clock.cc:648] HybridClock initialized: now 1781857296168823 us; error 48 us; skew 500 ppm
Jun 19 08:21:36 dist-test-slave-d3n8 krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857296, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:36.171877 19573 init.cc:377] Logged in from keytab as kudu/127.4.151.129@KRBTEST.COM (short username kudu)
I20260619 08:21:36.172464 19573 webserver.cc:460] Webserver started at http://127.4.151.129:44795/ using document root <none> and password file <none>
I20260619 08:21:36.172734 19573 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:36.172809 19573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:21:36.174387 19573 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:21:36.175211 19617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:36.175426 19573 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:21:36.175521 19573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "93af7b16aab148ec9d117f477a745284"
format_stamp: "Formatted at 2026-06-19 08:21:34 on dist-test-slave-d3n8"
server_key: "630de7b74e948fa46757e98c77d7bf23"
server_key_iv: "8819b7971ea9a1e5de7d1f546ea779fa"
server_key_version: "encryptionkey@0"
I20260619 08:21:36.175938 19573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:36.189790 19573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:36.192643 19573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:36.192858 19573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:36.193110 19573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:21:36.193658 19624 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 08:21:36.195670 19573 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 08:21:36.195763 19573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 08:21:36.195812 19573 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 08:21:36.196442 19573 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 08:21:36.196506 19573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 08:21:36.196573 19624 tablet_bootstrap.cc:492] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284: Bootstrap starting.
I20260619 08:21:36.203922 19573 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:33811
I20260619 08:21:36.203976 19731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:33811 every 8 connection(s)
I20260619 08:21:36.204403 19573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
I20260619 08:21:36.207897 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 19573
I20260619 08:21:36.208120 19624 log.cc:824] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284: Log is configured to *not* fsync() on all Append() calls
I20260619 08:21:36.208240 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:43707
--local_ip_for_outbound_sockets=127.4.151.130
--tserver_master_addrs=127.4.151.190:33625
--webserver_port=41893
--webserver_interface=127.4.151.130
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:21:36.211010 19624 tablet_bootstrap.cc:492] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:21:36.211330 19624 tablet_bootstrap.cc:492] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284: Bootstrap complete.
I20260619 08:21:36.212666 19624 ts_tablet_manager.cc:1397] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.013s
I20260619 08:21:36.214205 19624 raft_consensus.cc:348] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:36.214551 19624 raft_consensus.cc:729] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93af7b16aab148ec9d117f477a745284, State: Initialized, Role: FOLLOWER
I20260619 08:21:36.214779 19624 consensus_queue.cc:260] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:36.215065 19624 ts_tablet_manager.cc:1428] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260619 08:21:36.275103 19734 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:21:36.205095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.129:41217)
0619 08:21:36.205280 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:36.205325 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:36.205846 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:36.274858 (+ 69012us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":116,"threads_started":1}
W20260619 08:21:36.275458 19732 heartbeater.cc:644] Failed to heartbeat to 127.4.151.190:33625 (0 consecutive failures): Network error: Failed to ping master at 127.4.151.190:33625: Client connection negotiation failed: client connection to 127.4.151.190:33625: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 08:21:36.275846 19734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:36.275720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.129:50943)
0619 08:21:36.275782 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:36.275825 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 08:21:36.276218 19734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:36.276127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.129:57685)
0619 08:21:36.276180 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:36.276203 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 08:21:36.276347 19732 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 08:21:36.293483 19736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:36.293730 19736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:36.293771 19736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:36.295015 19736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:36.295091 19736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:36.295137 19736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:21:36.296682 19736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:43707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=41893
--tserver_master_addrs=127.4.151.190:33625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:36.296964 19736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:36.297209 19736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:36.302983 19743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:36.302978 19745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:36.302979 19742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:36.303285 19736 server_base.cc:1034] running on GCE node
I20260619 08:21:36.303534 19736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:36.303789 19736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:36.304982 19736 hybrid_clock.cc:648] HybridClock initialized: now 1781857296304960 us; error 51 us; skew 500 ppm
Jun 19 08:21:36 dist-test-slave-d3n8 krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857296, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:36.307996 19736 init.cc:377] Logged in from keytab as kudu/127.4.151.130@KRBTEST.COM (short username kudu)
I20260619 08:21:36.308569 19736 webserver.cc:460] Webserver started at http://127.4.151.130:41893/ using document root <none> and password file <none>
I20260619 08:21:36.308779 19736 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:36.308823 19736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:21:36.310334 19736 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:21:36.310935 19752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:36.311336 19736 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:21:36.311429 19736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "8b2473c351fe4d4a9be83937dbbc9f5b"
format_stamp: "Formatted at 2026-06-19 08:21:34 on dist-test-slave-d3n8"
server_key: "c83a18e6fa9c6720886437006150bddb"
server_key_iv: "70fd3c6974b6d49d2c471c88d85854e3"
server_key_version: "encryptionkey@0"
I20260619 08:21:36.311769 19736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:36.328509 19736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:36.331256 19736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:36.331393 19736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:36.331594 19736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:21:36.331925 19736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:21:36.331962 19736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:36.331986 19736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:21:36.331998 19736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:36.338382 19736 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:43707
I20260619 08:21:36.338454 19865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:43707 every 8 connection(s)
I20260619 08:21:36.338846 19736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
I20260619 08:21:36.340277 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:36.339615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.130:47899)
0619 08:21:36.339794 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:36.339876 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
W20260619 08:21:36.340631 19866 heartbeater.cc:644] Failed to heartbeat to 127.4.151.190:33625 (0 consecutive failures): Network error: Failed to ping master at 127.4.151.190:33625: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
I20260619 08:21:36.340993 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:36.340869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.130:53273)
0619 08:21:36.340922 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:36.340972 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 08:21:36.341310 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:36.341223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.130:57687)
0619 08:21:36.341270 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:36.341293 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 08:21:36.341579 19866 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 08:21:36.344771 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 19736
I20260619 08:21:36.345089 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:33519
--local_ip_for_outbound_sockets=127.4.151.131
--tserver_master_addrs=127.4.151.190:33625
--webserver_port=45601
--webserver_interface=127.4.151.131
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:21:36.432247 19869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:36.432462 19869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:36.432489 19869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:36.433736 19869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:36.433787 19869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:36.433815 19869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:21:36.435352 19869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:33519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=45601
--tserver_master_addrs=127.4.151.190:33625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:36.435624 19869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:36.435882 19869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:36.441399 19874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:36.441399 19875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:36.441416 19877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:36.441773 19869 server_base.cc:1034] running on GCE node
I20260619 08:21:36.441999 19869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:36.442210 19869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:36.443383 19869 hybrid_clock.cc:648] HybridClock initialized: now 1781857296443369 us; error 32 us; skew 500 ppm
Jun 19 08:21:36 dist-test-slave-d3n8 krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857296, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:36.446260 19869 init.cc:377] Logged in from keytab as kudu/127.4.151.131@KRBTEST.COM (short username kudu)
I20260619 08:21:36.446797 19869 webserver.cc:460] Webserver started at http://127.4.151.131:45601/ using document root <none> and password file <none>
I20260619 08:21:36.447058 19869 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:36.447140 19869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:21:36.448621 19869 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:21:36.449494 19884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:36.449749 19869 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 08:21:36.449869 19869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "d315c1374d844a9ab7774177f8150672"
format_stamp: "Formatted at 2026-06-19 08:21:34 on dist-test-slave-d3n8"
server_key: "793bdcbcef4d0dca95a2c22a17abc9c0"
server_key_iv: "ef676be221720c2b6e670f1e463e992c"
server_key_version: "encryptionkey@0"
I20260619 08:21:36.450377 19869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:36.462584 19869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:36.465229 19869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:36.465448 19869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:36.465727 19869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:21:36.466101 19869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:21:36.466182 19869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:36.466241 19869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:21:36.466300 19869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:36.472980 19869 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:33519
I20260619 08:21:36.473033 19997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:33519 every 8 connection(s)
I20260619 08:21:36.473445 19869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
I20260619 08:21:36.474538 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:36.474118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.131:34535)
0619 08:21:36.474307 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:36.474385 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":96,"threads_started":1}
W20260619 08:21:36.475013 19998 heartbeater.cc:644] Failed to heartbeat to 127.4.151.190:33625 (0 consecutive failures): Network error: Failed to ping master at 127.4.151.190:33625: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
I20260619 08:21:36.475353 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:36.475224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.131:46783)
0619 08:21:36.475298 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:36.475329 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 08:21:36.475648 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:36.475583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.131:48635)
0619 08:21:36.475609 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:36.475633 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260619 08:21:36.475766 19998 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 08:21:36.480978 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 19869
I20260619 08:21:37.277019 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:37.276760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.129:52321)
0619 08:21:37.276914 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:37.276954 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":66,"threads_started":1}
I20260619 08:21:37.342425 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:37.341981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.130:50483)
0619 08:21:37.342313 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:37.342355 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":261,"threads_started":1}
I20260619 08:21:37.476339 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:37.476090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.131:40511)
0619 08:21:37.476241 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:37.476277 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":61,"threads_started":1}
I20260619 08:21:37.806699 20005 raft_consensus.cc:482] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:21:37.806896 20005 raft_consensus.cc:504] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:37.807312 20005 leader_election.cc:288] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d315c1374d844a9ab7774177f8150672 (127.4.151.131:33519), 8b2473c351fe4d4a9be83937dbbc9f5b (127.4.151.130:43707)
Jun 19 08:21:37 dist-test-slave-d3n8 krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857296, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for kudu/127.4.151.130@KRBTEST.COM
Jun 19 08:21:37 dist-test-slave-d3n8 krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857296, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for kudu/127.4.151.131@KRBTEST.COM
I20260619 08:21:37.820156 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:37.807532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:33519 (local address 127.4.151.129:40745)
0619 08:21:37.807651 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:37.807693 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:37.807780 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:37.808733 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:37.808736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:37.809249 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:37.809431 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:37.809438 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:37.810597 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:37.810598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:37.811440 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:37.811443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:37.812163 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:37.812164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:37.812218 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:37.812957 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:37.812965 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:37.815452 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:37.818094 (+ 2642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:37.818095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:37.818095 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:37.818348 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:37.818709 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:37.818709 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:37.818710 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:37.818886 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:37.819461 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:37.819469 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:37.819643 (+ 174us) client_negotiation.cc:769] Sending connection context
0619 08:21:37.819792 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 08:21:37.820026 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":13,"thread_start_us":77,"threads_started":1}
I20260619 08:21:37.820290 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:37.807720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:43707 (local address 127.4.151.129:53139)
0619 08:21:37.807843 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:37.807851 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:37.807891 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:37.808734 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:37.808737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:37.809174 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:37.809431 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:37.809442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:37.810591 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:37.810592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:37.811411 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:37.811414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:37.812088 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:37.812089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:37.812204 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:37.812938 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:37.812947 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:37.814967 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:37.817855 (+ 2888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:37.817865 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:37.817869 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:37.818232 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:37.818709 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:37.818709 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:37.818710 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:37.818886 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:37.819467 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:37.819470 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:37.819642 (+ 172us) client_negotiation.cc:769] Sending connection context
0619 08:21:37.819792 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 08:21:37.820193 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":230,"thread_start_us":60,"threads_started":1}
I20260619 08:21:37.820664 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:37.807845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:53139 (local address 127.4.151.130:43707)
0619 08:21:37.807989 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:37.807996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:37.808076 (+ 80us) server_negotiation.cc:407] Connection header received
0619 08:21:37.808490 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:37.808492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:37.808529 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:37.808708 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:37.809569 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:37.810465 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:37.811541 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:37.811973 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:37.812118 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:37.815139 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:37.815148 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:37.815148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:37.815182 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:37.817756 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:37.818448 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:37.818452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:37.818452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:37.818594 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:37.818966 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:37.818967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:37.818968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:37.819376 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:37.819450 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:37.820029 (+ 579us) server_negotiation.cc:299] Negotiation successful
0619 08:21:37.820263 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":49,"threads_started":1}
I20260619 08:21:37.820647 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:37.807729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:40745 (local address 127.4.151.131:33519)
0619 08:21:37.807941 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:37.807955 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:37.807991 (+ 36us) server_negotiation.cc:407] Connection header received
0619 08:21:37.808468 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:37.808470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:37.808515 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:37.808682 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:37.809566 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:37.810474 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:37.811630 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:37.812040 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:37.812190 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:37.815605 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:37.815610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:37.815610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:37.815624 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:37.818003 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:37.818448 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:37.818452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:37.818452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:37.818594 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:37.818966 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:37.818967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:37.818968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:37.819323 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:37.819421 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:37.820029 (+ 608us) server_negotiation.cc:299] Negotiation successful
0619 08:21:37.820263 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":83,"threads_started":1}
W20260619 08:21:37.822214 19619 leader_election.cc:341] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b2473c351fe4d4a9be83937dbbc9f5b (127.4.151.130:43707): Not found: Tablet not found: 96ef5d62f07d4675a10a88cf6fd2029e
W20260619 08:21:37.822513 19619 leader_election.cc:341] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d315c1374d844a9ab7774177f8150672 (127.4.151.131:33519): Not found: Tablet not found: 96ef5d62f07d4675a10a88cf6fd2029e
I20260619 08:21:37.822628 19619 leader_election.cc:302] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93af7b16aab148ec9d117f477a745284; no voters: 8b2473c351fe4d4a9be83937dbbc9f5b, d315c1374d844a9ab7774177f8150672
I20260619 08:21:37.822773 20005 raft_consensus.cc:2726] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 08:21:38.277765 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:38.277594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.129:49319)
0619 08:21:38.277659 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:38.277697 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 08:21:38.343295 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:38.343031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.130:44179)
0619 08:21:38.343199 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:38.343246 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":64,"threads_started":1}
I20260619 08:21:38.477325 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:38.476962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.131:57343)
0619 08:21:38.477233 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:38.477278 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":174,"threads_started":1}
I20260619 08:21:39.278558 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:39.278299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.129:34065)
0619 08:21:39.278472 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:39.278513 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":61,"threads_started":1}
I20260619 08:21:39.344285 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:39.343953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.130:33709)
0619 08:21:39.344198 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:39.344239 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":163,"threads_started":1}
I20260619 08:21:39.478152 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:39.477915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.131:53111)
0619 08:21:39.478060 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:39.478105 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":59,"threads_started":1}
I20260619 08:21:39.899919 20015 raft_consensus.cc:482] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:21:39.900064 20015 raft_consensus.cc:504] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:39.900256 20015 leader_election.cc:288] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d315c1374d844a9ab7774177f8150672 (127.4.151.131:33519), 8b2473c351fe4d4a9be83937dbbc9f5b (127.4.151.130:43707)
W20260619 08:21:39.900785 19619 leader_election.cc:341] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b2473c351fe4d4a9be83937dbbc9f5b (127.4.151.130:43707): Not found: Tablet not found: 96ef5d62f07d4675a10a88cf6fd2029e
W20260619 08:21:39.900864 19619 leader_election.cc:341] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d315c1374d844a9ab7774177f8150672 (127.4.151.131:33519): Not found: Tablet not found: 96ef5d62f07d4675a10a88cf6fd2029e
I20260619 08:21:39.900915 19619 leader_election.cc:302] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93af7b16aab148ec9d117f477a745284; no voters: 8b2473c351fe4d4a9be83937dbbc9f5b, d315c1374d844a9ab7774177f8150672
I20260619 08:21:39.901018 20015 raft_consensus.cc:2726] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 08:21:40.279405 20016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:40.279099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.129:53823)
0619 08:21:40.279312 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:40.279356 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":64,"threads_started":1}
I20260619 08:21:40.345247 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:40.344902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.130:43507)
0619 08:21:40.345129 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:40.345174 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":68,"threads_started":1}
I20260619 08:21:40.479058 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:40.478785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.131:41735)
0619 08:21:40.478948 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:40.478990 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":76,"threads_started":1}
I20260619 08:21:41.280287 20019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:41.280071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.129:58667)
0619 08:21:41.280195 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:41.280237 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260619 08:21:41.346238 20020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:41.345950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.130:34555)
0619 08:21:41.346120 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:41.346174 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":69,"threads_started":1}
I20260619 08:21:41.441319 20021 raft_consensus.cc:482] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:21:41.441517 20021 raft_consensus.cc:504] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:41.441812 20021 leader_election.cc:288] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d315c1374d844a9ab7774177f8150672 (127.4.151.131:33519), 8b2473c351fe4d4a9be83937dbbc9f5b (127.4.151.130:43707)
W20260619 08:21:41.442389 19619 leader_election.cc:341] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d315c1374d844a9ab7774177f8150672 (127.4.151.131:33519): Not found: Tablet not found: 96ef5d62f07d4675a10a88cf6fd2029e
W20260619 08:21:41.442476 19619 leader_election.cc:341] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b2473c351fe4d4a9be83937dbbc9f5b (127.4.151.130:43707): Not found: Tablet not found: 96ef5d62f07d4675a10a88cf6fd2029e
I20260619 08:21:41.442493 19619 leader_election.cc:302] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93af7b16aab148ec9d117f477a745284; no voters: 8b2473c351fe4d4a9be83937dbbc9f5b, d315c1374d844a9ab7774177f8150672
I20260619 08:21:41.442574 20021 raft_consensus.cc:2726] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 08:21:41.480047 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:41.479742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.131:51345)
0619 08:21:41.479951 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:41.479999 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":137,"threads_started":1}
I20260619 08:21:42.281061 20023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:42.280805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.129:38463)
0619 08:21:42.280965 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:42.281005 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260619 08:21:42.347240 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:42.346962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.130:53555)
0619 08:21:42.347135 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:42.347190 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":71,"threads_started":1}
I20260619 08:21:42.480773 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:42.480530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.131:48773)
0619 08:21:42.480685 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:42.480724 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":74,"threads_started":1}
I20260619 08:21:43.281870 20026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:43.281616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.129:34249)
0619 08:21:43.281768 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:43.281822 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":62,"threads_started":1}
I20260619 08:21:43.348292 20027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:43.348014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.130:44747)
0619 08:21:43.348185 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:43.348240 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":80,"threads_started":1}
I20260619 08:21:43.404336 20028 raft_consensus.cc:482] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:21:43.404451 20028 raft_consensus.cc:504] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:43.404659 20028 leader_election.cc:288] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d315c1374d844a9ab7774177f8150672 (127.4.151.131:33519), 8b2473c351fe4d4a9be83937dbbc9f5b (127.4.151.130:43707)
W20260619 08:21:43.405084 19619 leader_election.cc:341] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b2473c351fe4d4a9be83937dbbc9f5b (127.4.151.130:43707): Not found: Tablet not found: 96ef5d62f07d4675a10a88cf6fd2029e
W20260619 08:21:43.405153 19619 leader_election.cc:341] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d315c1374d844a9ab7774177f8150672 (127.4.151.131:33519): Not found: Tablet not found: 96ef5d62f07d4675a10a88cf6fd2029e
I20260619 08:21:43.405169 19619 leader_election.cc:302] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93af7b16aab148ec9d117f477a745284; no voters: 8b2473c351fe4d4a9be83937dbbc9f5b, d315c1374d844a9ab7774177f8150672
I20260619 08:21:43.405313 20028 raft_consensus.cc:2726] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 08:21:43.481788 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:43.481528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.131:55519)
0619 08:21:43.481695 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:43.481738 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":74,"threads_started":1}
I20260619 08:21:44.282687 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:44.282435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.129:60967)
0619 08:21:44.282591 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:44.282638 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":63,"threads_started":1}
I20260619 08:21:44.349431 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:44.349016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.130:56185)
0619 08:21:44.349261 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:44.349305 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":169,"threads_started":1}
I20260619 08:21:44.482589 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:44.482354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.131:41189)
0619 08:21:44.482492 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:44.482534 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260619 08:21:45.283531 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:45.283290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.129:38935)
0619 08:21:45.283430 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:45.283476 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
I20260619 08:21:45.350201 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:45.349947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.130:56875)
0619 08:21:45.350100 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:45.350145 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260619 08:21:45.483286 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:45.483041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.131:60209)
0619 08:21:45.483204 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:45.483239 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":75,"threads_started":1}
I20260619 08:21:46.284394 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:46.284161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.129:57361)
0619 08:21:46.284304 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:46.284344 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":63,"threads_started":1}
I20260619 08:21:46.351202 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:46.350871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.130:34895)
0619 08:21:46.351026 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:46.351072 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":73,"threads_started":1}
I20260619 08:21:46.484051 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:46.483813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.131:53695)
0619 08:21:46.483953 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:46.483998 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
I20260619 08:21:47.285265 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:47.284949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.129:58073)
0619 08:21:47.285179 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:47.285217 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":75,"threads_started":1}
I20260619 08:21:47.352201 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:47.351940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.130:50473)
0619 08:21:47.352108 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:47.352153 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":80,"threads_started":1}
I20260619 08:21:47.484880 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:47.484626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.131:32975)
0619 08:21:47.484787 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:47.484832 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":60,"threads_started":1}
I20260619 08:21:47.553403 20042 raft_consensus.cc:482] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:21:47.553519 20042 raft_consensus.cc:504] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:47.553721 20042 leader_election.cc:288] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d315c1374d844a9ab7774177f8150672 (127.4.151.131:33519), 8b2473c351fe4d4a9be83937dbbc9f5b (127.4.151.130:43707)
W20260619 08:21:47.554243 19619 leader_election.cc:341] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d315c1374d844a9ab7774177f8150672 (127.4.151.131:33519): Not found: Tablet not found: 96ef5d62f07d4675a10a88cf6fd2029e
W20260619 08:21:47.554332 19619 leader_election.cc:341] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b2473c351fe4d4a9be83937dbbc9f5b (127.4.151.130:43707): Not found: Tablet not found: 96ef5d62f07d4675a10a88cf6fd2029e
I20260619 08:21:47.554371 19619 leader_election.cc:302] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93af7b16aab148ec9d117f477a745284; no voters: 8b2473c351fe4d4a9be83937dbbc9f5b, d315c1374d844a9ab7774177f8150672
I20260619 08:21:47.554450 20042 raft_consensus.cc:2726] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 08:21:48.286047 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:48.285816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.129:50441)
0619 08:21:48.285964 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:48.286001 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260619 08:21:48.353122 20044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:48.352784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.130:54177)
0619 08:21:48.352974 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:48.353074 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":107,"threads_started":1}
I20260619 08:21:48.485819 20045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:48.485549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.131:50067)
0619 08:21:48.485711 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:48.485770 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":82,"threads_started":1}
I20260619 08:21:49.286947 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:49.286684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.129:45237)
0619 08:21:49.286851 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:49.286898 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":66,"threads_started":1}
I20260619 08:21:49.353951 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:49.353727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.130:50917)
0619 08:21:49.353860 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:49.353904 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260619 08:21:49.357626 20048 raft_consensus.cc:482] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:21:49.357756 20048 raft_consensus.cc:504] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d315c1374d844a9ab7774177f8150672" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33519 } } peers { permanent_uuid: "93af7b16aab148ec9d117f477a745284" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 33811 } } peers { permanent_uuid: "8b2473c351fe4d4a9be83937dbbc9f5b" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 43707 } }
I20260619 08:21:49.357954 20048 leader_election.cc:288] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d315c1374d844a9ab7774177f8150672 (127.4.151.131:33519), 8b2473c351fe4d4a9be83937dbbc9f5b (127.4.151.130:43707)
W20260619 08:21:49.358496 19619 leader_election.cc:341] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d315c1374d844a9ab7774177f8150672 (127.4.151.131:33519): Not found: Tablet not found: 96ef5d62f07d4675a10a88cf6fd2029e
W20260619 08:21:49.358599 19619 leader_election.cc:341] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b2473c351fe4d4a9be83937dbbc9f5b (127.4.151.130:43707): Not found: Tablet not found: 96ef5d62f07d4675a10a88cf6fd2029e
I20260619 08:21:49.358664 19619 leader_election.cc:302] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93af7b16aab148ec9d117f477a745284; no voters: 8b2473c351fe4d4a9be83937dbbc9f5b, d315c1374d844a9ab7774177f8150672
I20260619 08:21:49.358783 20048 raft_consensus.cc:2726] T 96ef5d62f07d4675a10a88cf6fd2029e P 93af7b16aab148ec9d117f477a745284 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 08:21:49.486774 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:49.486499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.131:60643)
0619 08:21:49.486653 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:49.486702 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":67,"threads_started":1}
I20260619 08:21:50.287884 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:50.287662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.129:35805)
0619 08:21:50.287794 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:50.287835 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260619 08:21:50.354746 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:50.354519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.130:34179)
0619 08:21:50.354653 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:50.354699 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260619 08:21:50.487648 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:50.487403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.131:52149)
0619 08:21:50.487531 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:50.487569 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20260619 08:21:51.288740 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:51.288513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.129:32869)
0619 08:21:51.288647 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:51.288692 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":58,"threads_started":1}
I20260619 08:21:51.355587 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:51.355346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.130:36239)
0619 08:21:51.355493 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:51.355539 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":59,"threads_started":1}
I20260619 08:21:51.484488 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 19573
I20260619 08:21:51.488551 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:51.488310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:33625 (local address 127.4.151.131:44577)
0619 08:21:51.488442 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:51.488489 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260619 08:21:51.489574 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 19736
I20260619 08:21:51.494266 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 19869
2026-06-19T08:21:51Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18907 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:21:51 dist-test-slave-d3n8 krb5kdc[20058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:21:51 dist-test-slave-d3n8 krb5kdc[20058](info): set up 2 sockets
Jun 19 08:21:51 dist-test-slave-d3n8 krb5kdc[20058](info): commencing operation
krb5kdc: starting...
W20260619 08:21:52.557564 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:21:52 dist-test-slave-d3n8 krb5kdc[20058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:21:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:21:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:21:52.682395 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:42845
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:42845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:21:52.764122 20071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:52.764340 20071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:52.764364 20071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:52.765604 20071 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 08:21:52.765648 20071 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 08:21:52.765661 20071 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 08:21:52.765673 20071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:21:52.765686 20071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:52.765699 20071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:52.765712 20071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:21:52.765744 20071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:21:52.767378 20071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:42845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:42845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:52.767652 20071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:52.767887 20071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:21:52.773528 20071 server_base.cc:1034] running on GCE node
W20260619 08:21:52.773501 20077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:52.773509 20079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:52.773509 20076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:52.774371 20071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:52.774641 20071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:52.775822 20071 hybrid_clock.cc:648] HybridClock initialized: now 1781857312775811 us; error 70 us; skew 500 ppm
Jun 19 08:21:52 dist-test-slave-d3n8 krb5kdc[20058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857312, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:52.778553 20071 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:21:52.779006 20071 webserver.cc:460] Webserver started at http://127.4.151.190:40417/ using document root <none> and password file <none>
I20260619 08:21:52.779281 20071 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:52.779350 20071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:21:52.779510 20071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:21:52.780654 20071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "ac68371830c44d3788301ebc2b4e80ef"
format_stamp: "Formatted at 2026-06-19 08:21:52 on dist-test-slave-d3n8"
server_key: "ebe58aebf5b36e0bc728b142c3cb4d1f"
server_key_iv: "4e22cec072027b152c5fb9bd462ab652"
server_key_version: "encryptionkey@0"
I20260619 08:21:52.781028 20071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "ac68371830c44d3788301ebc2b4e80ef"
format_stamp: "Formatted at 2026-06-19 08:21:52 on dist-test-slave-d3n8"
server_key: "ebe58aebf5b36e0bc728b142c3cb4d1f"
server_key_iv: "4e22cec072027b152c5fb9bd462ab652"
server_key_version: "encryptionkey@0"
I20260619 08:21:52.782620 20071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:21:52.783526 20086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:52.783749 20071 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:21:52.783856 20071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "ac68371830c44d3788301ebc2b4e80ef"
format_stamp: "Formatted at 2026-06-19 08:21:52 on dist-test-slave-d3n8"
server_key: "ebe58aebf5b36e0bc728b142c3cb4d1f"
server_key_iv: "4e22cec072027b152c5fb9bd462ab652"
server_key_version: "encryptionkey@0"
I20260619 08:21:52.783947 20071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:52.805986 20071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:52.808391 20071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:52.808569 20071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:52.812625 20071 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:42845
I20260619 08:21:52.812669 20138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:42845 every 8 connection(s)
I20260619 08:21:52.813060 20071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:21:52.814059 20139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:21:52.816465 20139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef: Bootstrap starting.
I20260619 08:21:52.817109 20139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef: Neither blocks nor log segments found. Creating new log.
I20260619 08:21:52.817442 20139 log.cc:824] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef: Log is configured to *not* fsync() on all Append() calls
I20260619 08:21:52.817669 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 20071
I20260619 08:21:52.817811 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:21:52.818034 4702 external_mini_cluster.cc:1428] Setting key c1cfa0c1df994421ed029b68e9e16735
I20260619 08:21:52.818336 20139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef: No bootstrap required, opened a new log
I20260619 08:21:52.820134 20139 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac68371830c44d3788301ebc2b4e80ef" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42845 } }
I20260619 08:21:52.820415 20139 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:21:52.820482 20139 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac68371830c44d3788301ebc2b4e80ef, State: Initialized, Role: FOLLOWER
I20260619 08:21:52.820648 20139 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac68371830c44d3788301ebc2b4e80ef" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42845 } }
I20260619 08:21:52.820762 20139 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:21:52.820801 20139 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:21:52.820883 20139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:21:52.822017 20139 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac68371830c44d3788301ebc2b4e80ef" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42845 } }
I20260619 08:21:52.822110 20139 leader_election.cc:302] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac68371830c44d3788301ebc2b4e80ef; no voters:
I20260619 08:21:52.822329 20139 leader_election.cc:288] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:21:52.822358 20144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:21:52.822464 20144 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [term 1 LEADER]: Becoming Leader. State: Replica: ac68371830c44d3788301ebc2b4e80ef, State: Running, Role: LEADER
I20260619 08:21:52.822598 20139 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:21:52.822603 20144 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac68371830c44d3788301ebc2b4e80ef" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42845 } }
I20260619 08:21:52.823096 20146 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac68371830c44d3788301ebc2b4e80ef. Latest consensus state: current_term: 1 leader_uuid: "ac68371830c44d3788301ebc2b4e80ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac68371830c44d3788301ebc2b4e80ef" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42845 } } }
I20260619 08:21:52.823088 20145 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac68371830c44d3788301ebc2b4e80ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac68371830c44d3788301ebc2b4e80ef" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42845 } } }
I20260619 08:21:52.823259 20146 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [sys.catalog]: This master's current role is: LEADER
I20260619 08:21:52.823266 20145 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [sys.catalog]: This master's current role is: LEADER
I20260619 08:21:52.823614 20149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:21:52.824375 20149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:21:52.826278 20149 catalog_manager.cc:1347] Generated new cluster ID: e57fa568d99342e880ca166e9c4ded89
I20260619 08:21:52.826332 20149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:21:52 dist-test-slave-d3n8 krb5kdc[20058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:21:52.830977 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:52.818562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59500 (local address 127.4.151.190:42845)
0619 08:21:52.818746 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:52.818756 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:52.818789 (+ 33us) server_negotiation.cc:407] Connection header received
0619 08:21:52.820243 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:52.820255 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:52.820329 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:52.820502 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:52.821493 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:52.822667 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:52.823814 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:52.824431 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:52.824541 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:52.827511 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:52.827515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:52.827516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:52.827529 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:52.829294 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:52.829726 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:52.829729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:52.829729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:52.829842 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:52.830056 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:52.830057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:52.830058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:52.830319 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:52.830483 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:52.830581 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 08:21:52.830738 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260619 08:21:52.834764 20149 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:21:52.835469 20149 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:21:52.839190 20149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef: Generated new TSK 0
I20260619 08:21:52.839361 20149 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.151.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
I20260619 08:21:52.869743 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:42845
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:21:52.954536 20165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:52.954747 20165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:52.954772 20165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:52.955926 20165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:52.955962 20165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:52.955988 20165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:21:52.957429 20165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:42845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:52.957619 20165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:52.957803 20165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:52.963272 20171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:52.963413 20165 server_base.cc:1034] running on GCE node
W20260619 08:21:52.963289 20170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:52.963297 20173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:52.963811 20165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:52.964099 20165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:52.965293 20165 hybrid_clock.cc:648] HybridClock initialized: now 1781857312965305 us; error 75 us; skew 500 ppm
Jun 19 08:21:52 dist-test-slave-d3n8 krb5kdc[20058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857312, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:52.968254 20165 init.cc:377] Logged in from keytab as kudu/127.4.151.129@KRBTEST.COM (short username kudu)
I20260619 08:21:52.968798 20165 webserver.cc:460] Webserver started at http://127.4.151.129:41251/ using document root <none> and password file <none>
I20260619 08:21:52.969098 20165 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:52.969184 20165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:21:52.969337 20165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:21:52.970566 20165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "0d8f97fb52e048f79923c60a7e51c4cb"
format_stamp: "Formatted at 2026-06-19 08:21:52 on dist-test-slave-d3n8"
server_key: "03a8d733d06804c0d907afe8b26a8f1e"
server_key_iv: "eeda87f036f419bdeffc665a1c545bf1"
server_key_version: "encryptionkey@0"
I20260619 08:21:52.970925 20165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "0d8f97fb52e048f79923c60a7e51c4cb"
format_stamp: "Formatted at 2026-06-19 08:21:52 on dist-test-slave-d3n8"
server_key: "03a8d733d06804c0d907afe8b26a8f1e"
server_key_iv: "eeda87f036f419bdeffc665a1c545bf1"
server_key_version: "encryptionkey@0"
I20260619 08:21:52.972524 20165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:21:52.973490 20180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:52.973839 20165 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:21:52.973976 20165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "0d8f97fb52e048f79923c60a7e51c4cb"
format_stamp: "Formatted at 2026-06-19 08:21:52 on dist-test-slave-d3n8"
server_key: "03a8d733d06804c0d907afe8b26a8f1e"
server_key_iv: "eeda87f036f419bdeffc665a1c545bf1"
server_key_version: "encryptionkey@0"
I20260619 08:21:52.974174 20165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:53.025640 20165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:53.027999 20165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:53.028276 20165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:53.028555 20165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:21:53.028898 20165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:21:53.028955 20165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:53.028990 20165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:21:53.029019 20165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:53.035796 20165 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:32805
I20260619 08:21:53.035851 20293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:32805 every 8 connection(s)
I20260619 08:21:53.036207 20165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
Jun 19 08:21:53 dist-test-slave-d3n8 krb5kdc[20058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857312, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:21:53.045975 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 20165
I20260619 08:21:53.046099 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:21:53.046315 4702 external_mini_cluster.cc:1428] Setting key 2982fd19fa422eeaf32d85c29840a534
I20260619 08:21:53.046864 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:53.037061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:46713 (local address 127.4.151.190:42845)
0619 08:21:53.037141 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:53.037143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:53.037357 (+ 214us) server_negotiation.cc:407] Connection header received
0619 08:21:53.037929 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:53.037930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:53.037941 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:53.038020 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:53.039117 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.039649 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.040694 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.040921 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.040992 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:53.043436 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:53.043438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:53.043438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:53.043449 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:53.045081 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:53.045660 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:53.045661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:53.045661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:53.045719 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:53.046036 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:53.046037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:53.046037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:53.046214 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:53.046281 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:53.046635 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 08:21:53.046765 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 08:21:53.047052 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:53.037010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.129:46713)
0619 08:21:53.037188 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:53.037239 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:53.037809 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:53.038089 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:53.038093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:53.038749 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:53.038985 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:53.038989 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.039755 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.039755 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:53.040595 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:53.040597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.041011 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.041012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:53.041127 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:53.041866 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:53.041874 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:53.043315 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:53.045244 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:53.045252 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:53.045254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:53.045572 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:53.045798 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:53.045799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:53.045799 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:53.045955 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:53.046269 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:53.046276 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:53.046468 (+ 192us) client_negotiation.cc:769] Sending connection context
0619 08:21:53.046573 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 08:21:53.046778 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":102,"threads_started":1}
I20260619 08:21:53.047876 20294 heartbeater.cc:347] Connected to a master server at 127.4.151.190:42845
I20260619 08:21:53.047994 20294 heartbeater.cc:464] Registering TS with master...
I20260619 08:21:53.048192 20294 heartbeater.cc:511] Master 127.4.151.190:42845 requested a full tablet report, sending...
I20260619 08:21:53.048774 20103 ts_manager.cc:194] Registered new tserver with Master: 0d8f97fb52e048f79923c60a7e51c4cb (127.4.151.129:32805)
I20260619 08:21:53.049584 20103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.129@KRBTEST.COM'} at 127.4.151.129:46713
WARNING: no policy specified for kudu/127.4.151.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
I20260619 08:21:53.078015 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:42845
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:21:53.160576 20299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:53.160830 20299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:53.160872 20299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:53.162168 20299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:53.162245 20299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:53.162297 20299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:21:53.163801 20299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:42845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:53.164110 20299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:53.164353 20299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:53.169615 20304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:53.169780 20307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:53.169785 20305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:53.169986 20299 server_base.cc:1034] running on GCE node
I20260619 08:21:53.170173 20299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:53.170387 20299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:53.171636 20299 hybrid_clock.cc:648] HybridClock initialized: now 1781857313171600 us; error 42 us; skew 500 ppm
Jun 19 08:21:53 dist-test-slave-d3n8 krb5kdc[20058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857313, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:53.174366 20299 init.cc:377] Logged in from keytab as kudu/127.4.151.130@KRBTEST.COM (short username kudu)
I20260619 08:21:53.174821 20299 webserver.cc:460] Webserver started at http://127.4.151.130:44915/ using document root <none> and password file <none>
I20260619 08:21:53.175060 20299 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:53.175130 20299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:21:53.175292 20299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:21:53.176381 20299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a"
format_stamp: "Formatted at 2026-06-19 08:21:53 on dist-test-slave-d3n8"
server_key: "57bdb51ffc163a93e812d327133fa0cd"
server_key_iv: "0fbd96f440d5fa646955ee46d157b44f"
server_key_version: "encryptionkey@0"
I20260619 08:21:53.176754 20299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a"
format_stamp: "Formatted at 2026-06-19 08:21:53 on dist-test-slave-d3n8"
server_key: "57bdb51ffc163a93e812d327133fa0cd"
server_key_iv: "0fbd96f440d5fa646955ee46d157b44f"
server_key_version: "encryptionkey@0"
I20260619 08:21:53.178252 20299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:21:53.179165 20314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:53.179389 20299 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:21:53.179494 20299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a"
format_stamp: "Formatted at 2026-06-19 08:21:53 on dist-test-slave-d3n8"
server_key: "57bdb51ffc163a93e812d327133fa0cd"
server_key_iv: "0fbd96f440d5fa646955ee46d157b44f"
server_key_version: "encryptionkey@0"
I20260619 08:21:53.179582 20299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:53.196684 20299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:53.198873 20299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:53.199041 20299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:53.199293 20299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:21:53.199616 20299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:21:53.199677 20299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:53.199734 20299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:21:53.199781 20299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:53.206002 20299 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:42405
I20260619 08:21:53.206104 20427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:42405 every 8 connection(s)
I20260619 08:21:53.206416 20299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
Jun 19 08:21:53 dist-test-slave-d3n8 krb5kdc[20058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857313, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:21:53.213562 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 20299
I20260619 08:21:53.213653 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:21:53.213862 4702 external_mini_cluster.cc:1428] Setting key 7d979f35d63c10b9c238f90d39158ae7
I20260619 08:21:53.216581 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:53.207290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:38259 (local address 127.4.151.190:42845)
0619 08:21:53.207357 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:53.207359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:53.207611 (+ 252us) server_negotiation.cc:407] Connection header received
0619 08:21:53.208159 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:53.208160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:53.208173 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:53.208223 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:53.209172 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.209778 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.210793 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.211006 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.211093 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:53.213548 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:53.213552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:53.213552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:53.213564 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:53.215138 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:53.215588 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:53.215589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:53.215590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:53.215649 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:53.215908 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:53.215909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:53.215909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:53.216030 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:53.216110 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:53.216393 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 08:21:53.216490 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 08:21:53.216784 20430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:53.207281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.130:38259)
0619 08:21:53.207469 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:53.207506 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:53.208047 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:53.208297 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:53.208301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:53.208877 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:53.209069 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:53.209073 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.209884 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.209885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:53.210684 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:53.210687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.211102 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.211103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:53.211245 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:53.211801 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:53.211808 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:53.213429 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:53.215251 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:53.215258 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:53.215261 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:53.215494 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:53.215723 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:53.215724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:53.215724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:53.215839 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:53.216099 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:53.216107 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:53.216259 (+ 152us) client_negotiation.cc:769] Sending connection context
0619 08:21:53.216373 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 08:21:53.216535 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":97,"threads_started":1}
I20260619 08:21:53.217322 20428 heartbeater.cc:347] Connected to a master server at 127.4.151.190:42845
I20260619 08:21:53.217459 20428 heartbeater.cc:464] Registering TS with master...
I20260619 08:21:53.217679 20428 heartbeater.cc:511] Master 127.4.151.190:42845 requested a full tablet report, sending...
I20260619 08:21:53.218132 20103 ts_manager.cc:194] Registered new tserver with Master: aa44be97d3d24fc2b1ff2ea93acf6f9a (127.4.151.130:42405)
I20260619 08:21:53.218683 20103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.130@KRBTEST.COM'} at 127.4.151.130:38259
WARNING: no policy specified for kudu/127.4.151.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
I20260619 08:21:53.245450 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:42845
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:21:53.330320 20433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:21:53.330596 20433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:21:53.330641 20433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:21:53.332003 20433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:21:53.332082 20433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:21:53.332151 20433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:21:53.333899 20433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:42845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:21:53.334228 20433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:21:53.334486 20433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:21:53.340103 20441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:53.340164 20438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:21:53.340193 20439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:21:53.340378 20433 server_base.cc:1034] running on GCE node
I20260619 08:21:53.340595 20433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:21:53.340811 20433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:21:53.342003 20433 hybrid_clock.cc:648] HybridClock initialized: now 1781857313341984 us; error 35 us; skew 500 ppm
Jun 19 08:21:53 dist-test-slave-d3n8 krb5kdc[20058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857313, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:21:53.344755 20433 init.cc:377] Logged in from keytab as kudu/127.4.151.131@KRBTEST.COM (short username kudu)
I20260619 08:21:53.345224 20433 webserver.cc:460] Webserver started at http://127.4.151.131:40871/ using document root <none> and password file <none>
I20260619 08:21:53.345534 20433 fs_manager.cc:359] Metadata directory not provided
I20260619 08:21:53.345609 20433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:21:53.345804 20433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:21:53.346964 20433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "1b64c563f60d4fb48895cbaaad57a82d"
format_stamp: "Formatted at 2026-06-19 08:21:53 on dist-test-slave-d3n8"
server_key: "f208fd339cf2bd3bb06eafea1ac11054"
server_key_iv: "eb4523c764e3871f9845a418035dba63"
server_key_version: "encryptionkey@0"
I20260619 08:21:53.347355 20433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "1b64c563f60d4fb48895cbaaad57a82d"
format_stamp: "Formatted at 2026-06-19 08:21:53 on dist-test-slave-d3n8"
server_key: "f208fd339cf2bd3bb06eafea1ac11054"
server_key_iv: "eb4523c764e3871f9845a418035dba63"
server_key_version: "encryptionkey@0"
I20260619 08:21:53.348812 20433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:21:53.349876 20448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:53.350276 20433 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 08:21:53.350432 20433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "1b64c563f60d4fb48895cbaaad57a82d"
format_stamp: "Formatted at 2026-06-19 08:21:53 on dist-test-slave-d3n8"
server_key: "f208fd339cf2bd3bb06eafea1ac11054"
server_key_iv: "eb4523c764e3871f9845a418035dba63"
server_key_version: "encryptionkey@0"
I20260619 08:21:53.350520 20433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:21:53.389279 20433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:21:53.391775 20433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:21:53.391949 20433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:21:53.392203 20433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:21:53.392529 20433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:21:53.392583 20433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:53.392625 20433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:21:53.392670 20433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:21:53.398643 20433 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:42699
I20260619 08:21:53.398689 20561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:42699 every 8 connection(s)
I20260619 08:21:53.399046 20433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
I20260619 08:21:53.401319 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 20433
I20260619 08:21:53.401484 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:21:53.401703 4702 external_mini_cluster.cc:1428] Setting key d822d719b6d897119a4485c030eb3a7e
Jun 19 08:21:53 dist-test-slave-d3n8 krb5kdc[20058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857313, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:21:53.409505 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:53.399769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:36625 (local address 127.4.151.190:42845)
0619 08:21:53.399858 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:53.399860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:53.400036 (+ 176us) server_negotiation.cc:407] Connection header received
0619 08:21:53.400612 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:53.400614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:53.400624 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:53.400700 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:53.401815 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.402610 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.403656 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.403870 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.403930 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:53.406430 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:53.406431 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:53.406431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:53.406438 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:53.408006 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:53.408451 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:53.408452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:53.408452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:53.408515 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:53.408841 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:53.408842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:53.408842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:53.408967 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:53.409031 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:53.409302 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 08:21:53.409369 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 08:21:53.409741 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:53.399735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.131:36625)
0619 08:21:53.399912 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:53.399949 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:53.400508 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:53.400758 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:53.400762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:53.401430 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:21:53.401716 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:53.401722 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.402722 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.402723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:53.403565 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:53.403567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.403967 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.403968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:53.404103 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:53.404624 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:21:53.404632 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:21:53.406328 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:21:53.408097 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:53.408105 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:53.408108 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:53.408369 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:53.408609 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:21:53.408610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:21:53.408610 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:21:53.408753 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:21:53.409043 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:21:53.409048 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:21:53.409170 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 08:21:53.409270 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 08:21:53.409493 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":101,"threads_started":1}
I20260619 08:21:53.410236 20562 heartbeater.cc:347] Connected to a master server at 127.4.151.190:42845
I20260619 08:21:53.410343 20562 heartbeater.cc:464] Registering TS with master...
I20260619 08:21:53.410543 20562 heartbeater.cc:511] Master 127.4.151.190:42845 requested a full tablet report, sending...
I20260619 08:21:53.410909 20103 ts_manager.cc:194] Registered new tserver with Master: 1b64c563f60d4fb48895cbaaad57a82d (127.4.151.131:42699)
I20260619 08:21:53.411267 20103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.131@KRBTEST.COM'} at 127.4.151.131:36625
I20260619 08:21:53.413529 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:21:53.421368 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:53.414877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59502 (local address 127.4.151.190:42845)
0619 08:21:53.414938 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:53.414940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:53.415151 (+ 211us) server_negotiation.cc:407] Connection header received
0619 08:21:53.415175 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:53.415175 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:53.415184 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:53.415233 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:21:53.415749 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.416297 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.417051 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.417276 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.417340 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:53.418775 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:21:53.418776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:21:53.418777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:21:53.418783 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:21:53.420269 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:53.420600 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:53.420601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:53.420601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:53.420649 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:21:53.420835 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:21:53.420835 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:21:53.420835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:21:53.420974 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:21:53.421123 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:53.421207 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 08:21:53.421270 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 08:21:53.423343 20103 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:21:53.425138 20103 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:21:53.431985 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:53.427733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:42699 (local address 127.0.0.1:54352)
0619 08:21:53.428110 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:53.428122 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:53.428168 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:53.428424 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:53.428425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:53.428427 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:21:53.428473 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:53.428474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.429752 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.429752 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:53.430832 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:53.430835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.431683 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.431684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:53.431760 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:53.431774 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 08:21:53.431849 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 08:21:53.431902 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":71,"threads_started":1}
I20260619 08:21:53.432030 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:53.427944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54352 (local address 127.4.151.131:42699)
0619 08:21:53.428108 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:53.428112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:53.428225 (+ 113us) server_negotiation.cc:407] Connection header received
0619 08:21:53.428271 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:53.428273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:53.428287 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:53.428355 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:21:53.428550 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.429631 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.430921 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.431601 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.431684 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:53.431743 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:53.431876 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 08:21:53.431919 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260619 08:21:53.432340 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:53.427522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:32805 (local address 127.0.0.1:39998)
0619 08:21:53.427817 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:53.427861 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:53.427932 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:53.428185 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:53.428188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:53.428190 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:21:53.428263 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:53.428265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.429746 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.429747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:53.431107 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:53.431110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.432098 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.432099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:53.432169 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:53.432173 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:21:53.432226 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 08:21:53.432262 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":35,"thread_start_us":84,"threads_started":1}
I20260619 08:21:53.432399 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:53.427606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39998 (local address 127.4.151.129:32805)
0619 08:21:53.427799 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:53.427804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:53.428028 (+ 224us) server_negotiation.cc:407] Connection header received
0619 08:21:53.428074 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:53.428077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:53.428094 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:53.428179 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:21:53.428418 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.429623 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.431198 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.431942 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.432008 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:53.432058 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:53.432245 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 08:21:53.432295 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":102,"threads_started":1}
I20260619 08:21:53.432888 20573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:53.427517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:42405 (local address 127.0.0.1:42142)
0619 08:21:53.427808 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:53.427860 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:53.427931 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:53.428198 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:53.428199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:53.428200 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:21:53.428266 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:53.428268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.430643 (+ 2375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.430644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:53.431520 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:53.431522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.432637 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.432638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:53.432704 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:53.432707 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:21:53.432780 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 08:21:53.432820 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":3,"thread_start_us":58,"threads_started":1}
I20260619 08:21:53.432940 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:53.427630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42142 (local address 127.4.151.130:42405)
0619 08:21:53.427890 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:53.427894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:53.427907 (+ 13us) server_negotiation.cc:407] Connection header received
0619 08:21:53.428016 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:53.428018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:53.428033 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:53.428109 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:21:53.429455 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.430538 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.431599 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.432307 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.432371 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:53.432420 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:53.432808 (+ 388us) server_negotiation.cc:299] Negotiation successful
0619 08:21:53.432852 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":87,"threads_started":1}
I20260619 08:21:53.433207 20228 tablet_service.cc:1467] Processing CreateTablet for tablet c59b15be1be14ed1a320fe09ad43f6bc (DEFAULT_TABLE table=test-table [id=7c3edb96e70048308955e8e84952fd31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:21:53.433626 20228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c59b15be1be14ed1a320fe09ad43f6bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:21:53.433576 20362 tablet_service.cc:1467] Processing CreateTablet for tablet c59b15be1be14ed1a320fe09ad43f6bc (DEFAULT_TABLE table=test-table [id=7c3edb96e70048308955e8e84952fd31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:21:53.433887 20362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c59b15be1be14ed1a320fe09ad43f6bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:21:53.433825 20496 tablet_service.cc:1467] Processing CreateTablet for tablet c59b15be1be14ed1a320fe09ad43f6bc (DEFAULT_TABLE table=test-table [id=7c3edb96e70048308955e8e84952fd31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:21:53.434118 20496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c59b15be1be14ed1a320fe09ad43f6bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:21:53.436542 20579 tablet_bootstrap.cc:492] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb: Bootstrap starting.
I20260619 08:21:53.437223 20579 tablet_bootstrap.cc:654] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb: Neither blocks nor log segments found. Creating new log.
I20260619 08:21:53.437204 20580 tablet_bootstrap.cc:492] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d: Bootstrap starting.
I20260619 08:21:53.437889 20579 log.cc:824] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb: Log is configured to *not* fsync() on all Append() calls
I20260619 08:21:53.438067 20580 tablet_bootstrap.cc:654] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d: Neither blocks nor log segments found. Creating new log.
I20260619 08:21:53.438668 20580 log.cc:824] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d: Log is configured to *not* fsync() on all Append() calls
I20260619 08:21:53.438798 20579 tablet_bootstrap.cc:492] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb: No bootstrap required, opened a new log
I20260619 08:21:53.438865 20579 ts_tablet_manager.cc:1397] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 08:21:53.439025 20581 tablet_bootstrap.cc:492] T c59b15be1be14ed1a320fe09ad43f6bc P aa44be97d3d24fc2b1ff2ea93acf6f9a: Bootstrap starting.
I20260619 08:21:53.439493 20580 tablet_bootstrap.cc:492] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d: No bootstrap required, opened a new log
I20260619 08:21:53.439582 20580 ts_tablet_manager.cc:1397] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:21:53.439716 20581 tablet_bootstrap.cc:654] T c59b15be1be14ed1a320fe09ad43f6bc P aa44be97d3d24fc2b1ff2ea93acf6f9a: Neither blocks nor log segments found. Creating new log.
I20260619 08:21:53.440249 20581 log.cc:824] T c59b15be1be14ed1a320fe09ad43f6bc P aa44be97d3d24fc2b1ff2ea93acf6f9a: Log is configured to *not* fsync() on all Append() calls
I20260619 08:21:53.441071 20581 tablet_bootstrap.cc:492] T c59b15be1be14ed1a320fe09ad43f6bc P aa44be97d3d24fc2b1ff2ea93acf6f9a: No bootstrap required, opened a new log
I20260619 08:21:53.441143 20581 ts_tablet_manager.cc:1397] T c59b15be1be14ed1a320fe09ad43f6bc P aa44be97d3d24fc2b1ff2ea93acf6f9a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:21:53.441330 20579 raft_consensus.cc:348] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } }
I20260619 08:21:53.441556 20579 raft_consensus.cc:374] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:21:53.441593 20579 raft_consensus.cc:729] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d8f97fb52e048f79923c60a7e51c4cb, State: Initialized, Role: FOLLOWER
I20260619 08:21:53.441332 20580 raft_consensus.cc:348] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } }
I20260619 08:21:53.441636 20580 raft_consensus.cc:374] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:21:53.441695 20580 raft_consensus.cc:729] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b64c563f60d4fb48895cbaaad57a82d, State: Initialized, Role: FOLLOWER
I20260619 08:21:53.441740 20579 consensus_queue.cc:260] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } }
I20260619 08:21:53.441805 20580 consensus_queue.cc:260] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } }
I20260619 08:21:53.442024 20579 ts_tablet_manager.cc:1428] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260619 08:21:53.442085 20580 ts_tablet_manager.cc:1428] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260619 08:21:53.442147 20294 heartbeater.cc:503] Master 127.4.151.190:42845 was elected leader, sending a full tablet report...
I20260619 08:21:53.442217 20562 heartbeater.cc:503] Master 127.4.151.190:42845 was elected leader, sending a full tablet report...
I20260619 08:21:53.443259 20581 raft_consensus.cc:348] T c59b15be1be14ed1a320fe09ad43f6bc P aa44be97d3d24fc2b1ff2ea93acf6f9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } }
I20260619 08:21:53.443437 20581 raft_consensus.cc:374] T c59b15be1be14ed1a320fe09ad43f6bc P aa44be97d3d24fc2b1ff2ea93acf6f9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:21:53.443478 20581 raft_consensus.cc:729] T c59b15be1be14ed1a320fe09ad43f6bc P aa44be97d3d24fc2b1ff2ea93acf6f9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa44be97d3d24fc2b1ff2ea93acf6f9a, State: Initialized, Role: FOLLOWER
I20260619 08:21:53.443598 20581 consensus_queue.cc:260] T c59b15be1be14ed1a320fe09ad43f6bc P aa44be97d3d24fc2b1ff2ea93acf6f9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } }
I20260619 08:21:53.443912 20581 ts_tablet_manager.cc:1428] T c59b15be1be14ed1a320fe09ad43f6bc P aa44be97d3d24fc2b1ff2ea93acf6f9a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:21:53.444008 20428 heartbeater.cc:503] Master 127.4.151.190:42845 was elected leader, sending a full tablet report...
W20260619 08:21:53.456992 20429 tablet.cc:2369] T c59b15be1be14ed1a320fe09ad43f6bc P aa44be97d3d24fc2b1ff2ea93acf6f9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:21:53.536996 20295 tablet.cc:2369] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:21:53.610988 20586 raft_consensus.cc:482] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:21:53.611129 20586 raft_consensus.cc:504] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } }
I20260619 08:21:53.611461 20586 leader_election.cc:288] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d8f97fb52e048f79923c60a7e51c4cb (127.4.151.129:32805), aa44be97d3d24fc2b1ff2ea93acf6f9a (127.4.151.130:42405)
I20260619 08:21:53.615140 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:53.611702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:32805 (local address 127.4.151.131:53373)
0619 08:21:53.611745 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:53.611771 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:53.611820 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:53.612051 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:53.612052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:53.612055 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:21:53.612303 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:53.612306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.613032 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.613033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:53.614295 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:53.614298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.614865 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.614866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:53.614951 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:53.614954 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:21:53.615021 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 08:21:53.615061 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 08:21:53.615140 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:53.611848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:53373 (local address 127.4.151.129:32805)
0619 08:21:53.611907 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:53.611909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:53.611916 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:21:53.611941 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:53.611941 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:53.611949 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:53.612039 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:21:53.612441 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.612935 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.614393 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.614770 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.614867 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:53.614877 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:53.615028 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 08:21:53.615068 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 08:21:53.615615 20248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59b15be1be14ed1a320fe09ad43f6bc" candidate_uuid: "1b64c563f60d4fb48895cbaaad57a82d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" is_pre_election: true
I20260619 08:21:53.615777 20248 raft_consensus.cc:2454] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b64c563f60d4fb48895cbaaad57a82d in term 0.
I20260619 08:21:53.615907 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:53.611773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:48407 (local address 127.4.151.130:42405)
0619 08:21:53.611818 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:53.611820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:53.612269 (+ 449us) server_negotiation.cc:407] Connection header received
0619 08:21:53.612305 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:53.612306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:53.612318 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:53.612385 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:21:53.613187 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.614030 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.614960 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.615548 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.615640 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:53.615648 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:53.615806 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 08:21:53.615842 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:21:53.615921 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:53.611702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:42405 (local address 127.4.151.131:48407)
0619 08:21:53.612070 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:53.612088 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 08:21:53.612143 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:21:53.613029 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:21:53.613030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:21:53.613031 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:21:53.613097 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:53.613098 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.614106 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.614106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:53.614854 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:21:53.614856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.615640 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.615641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:21:53.615739 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:53.615744 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:21:53.615800 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 08:21:53.615840 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":192,"thread_start_us":121,"threads_started":1}
I20260619 08:21:53.616060 20449 leader_election.cc:302] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d8f97fb52e048f79923c60a7e51c4cb, 1b64c563f60d4fb48895cbaaad57a82d; no voters:
I20260619 08:21:53.616187 20586 raft_consensus.cc:2781] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:21:53.616238 20586 raft_consensus.cc:482] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:21:53.616268 20586 raft_consensus.cc:3037] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:21:53.616298 20382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59b15be1be14ed1a320fe09ad43f6bc" candidate_uuid: "1b64c563f60d4fb48895cbaaad57a82d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" is_pre_election: true
I20260619 08:21:53.616463 20382 raft_consensus.cc:2454] T c59b15be1be14ed1a320fe09ad43f6bc P aa44be97d3d24fc2b1ff2ea93acf6f9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b64c563f60d4fb48895cbaaad57a82d in term 0.
I20260619 08:21:53.617148 20586 raft_consensus.cc:504] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } }
I20260619 08:21:53.617293 20586 leader_election.cc:288] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d [CANDIDATE]: Term 1 election: Requested vote from peers 0d8f97fb52e048f79923c60a7e51c4cb (127.4.151.129:32805), aa44be97d3d24fc2b1ff2ea93acf6f9a (127.4.151.130:42405)
I20260619 08:21:53.617496 20382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59b15be1be14ed1a320fe09ad43f6bc" candidate_uuid: "1b64c563f60d4fb48895cbaaad57a82d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a"
I20260619 08:21:53.617512 20248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59b15be1be14ed1a320fe09ad43f6bc" candidate_uuid: "1b64c563f60d4fb48895cbaaad57a82d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d8f97fb52e048f79923c60a7e51c4cb"
I20260619 08:21:53.617585 20382 raft_consensus.cc:3037] T c59b15be1be14ed1a320fe09ad43f6bc P aa44be97d3d24fc2b1ff2ea93acf6f9a [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:21:53.617585 20248 raft_consensus.cc:3037] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:21:53.618466 20382 raft_consensus.cc:2454] T c59b15be1be14ed1a320fe09ad43f6bc P aa44be97d3d24fc2b1ff2ea93acf6f9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b64c563f60d4fb48895cbaaad57a82d in term 1.
I20260619 08:21:53.618466 20248 raft_consensus.cc:2454] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b64c563f60d4fb48895cbaaad57a82d in term 1.
I20260619 08:21:53.618709 20449 leader_election.cc:302] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d8f97fb52e048f79923c60a7e51c4cb, 1b64c563f60d4fb48895cbaaad57a82d; no voters:
I20260619 08:21:53.618813 20586 raft_consensus.cc:2781] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:21:53.618975 20586 raft_consensus.cc:686] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d [term 1 LEADER]: Becoming Leader. State: Replica: 1b64c563f60d4fb48895cbaaad57a82d, State: Running, Role: LEADER
I20260619 08:21:53.619081 20586 consensus_queue.cc:237] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } }
I20260619 08:21:53.619973 20102 catalog_manager.cc:5495] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b64c563f60d4fb48895cbaaad57a82d (127.4.151.131). New cstate: current_term: 1 leader_uuid: "1b64c563f60d4fb48895cbaaad57a82d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } health_report { overall_health: HEALTHY } } }
W20260619 08:21:53.649721 20563 tablet.cc:2369] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:21:53.660051 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:53.657429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54358 (local address 127.4.151.131:42699)
0619 08:21:53.657498 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 08:21:53.657503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:21:53.657513 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:21:53.657573 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:21:53.657574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:21:53.657591 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:21:53.657656 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:21:53.657819 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.658311 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.659199 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:21:53.659428 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:21:53.659484 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:21:53.659706 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:21:53.659814 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:21:53.659892 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 08:21:53.659953 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 08:21:53.659972 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 08:21:53.661948 20382 raft_consensus.cc:1261] T c59b15be1be14ed1a320fe09ad43f6bc P aa44be97d3d24fc2b1ff2ea93acf6f9a [term 1 FOLLOWER]: Refusing update from remote peer 1b64c563f60d4fb48895cbaaad57a82d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:21:53.661991 20248 raft_consensus.cc:1261] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb [term 1 FOLLOWER]: Refusing update from remote peer 1b64c563f60d4fb48895cbaaad57a82d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:21:53.662384 20586 consensus_queue.cc:1035] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:21:53.662552 20589 consensus_queue.cc:1035] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:21:53.665360 20592 mvcc.cc:205] Tried to move back new op lower bound from 7298487556756062208 to 7298487556584480768. Current Snapshot: MvccSnapshot[applied={T|T < 7298487556756062208}]
I20260619 08:21:53.665474 20593 mvcc.cc:205] Tried to move back new op lower bound from 7298487556756062208 to 7298487556584480768. Current Snapshot: MvccSnapshot[applied={T|T < 7298487556756062208}]
I20260619 08:21:53.665722 20594 mvcc.cc:205] Tried to move back new op lower bound from 7298487556756062208 to 7298487556584480768. Current Snapshot: MvccSnapshot[applied={T|T < 7298487556756062208}]
I20260619 08:21:53.668524 20102 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:21:53.668614 20102 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:21:53.669447 20102 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef: Sending DeleteTablet for 3 replicas of tablet c59b15be1be14ed1a320fe09ad43f6bc
I20260619 08:21:53.669812 20228 tablet_service.cc:1514] Processing DeleteTablet for tablet c59b15be1be14ed1a320fe09ad43f6bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:21:53 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:39998
I20260619 08:21:53.669824 20496 tablet_service.cc:1514] Processing DeleteTablet for tablet c59b15be1be14ed1a320fe09ad43f6bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:21:53 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:54352
I20260619 08:21:53.669879 20362 tablet_service.cc:1514] Processing DeleteTablet for tablet c59b15be1be14ed1a320fe09ad43f6bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:21:53 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:42142
I20260619 08:21:53.670081 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 20165
I20260619 08:21:53.671105 20602 tablet_replica.cc:330] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d: stopping tablet replica
I20260619 08:21:53.671236 20602 raft_consensus.cc:2229] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:21:53.671372 20602 raft_consensus.cc:2258] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:21:53.671532 20604 tablet_replica.cc:330] T c59b15be1be14ed1a320fe09ad43f6bc P aa44be97d3d24fc2b1ff2ea93acf6f9a: stopping tablet replica
I20260619 08:21:53.671638 20604 raft_consensus.cc:2229] T c59b15be1be14ed1a320fe09ad43f6bc P aa44be97d3d24fc2b1ff2ea93acf6f9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:21:53.671759 20604 raft_consensus.cc:2258] T c59b15be1be14ed1a320fe09ad43f6bc P aa44be97d3d24fc2b1ff2ea93acf6f9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:21:53.672192 20602 ts_tablet_manager.cc:1905] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:21:53.673835 20604 ts_tablet_manager.cc:1905] T c59b15be1be14ed1a320fe09ad43f6bc P aa44be97d3d24fc2b1ff2ea93acf6f9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:21:53.674116 20602 ts_tablet_manager.cc:1918] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:21:53.674202 20602 log.cc:1196] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/c59b15be1be14ed1a320fe09ad43f6bc
I20260619 08:21:53.674533 20602 ts_tablet_manager.cc:1939] T c59b15be1be14ed1a320fe09ad43f6bc P 1b64c563f60d4fb48895cbaaad57a82d: Deleting consensus metadata
I20260619 08:21:53.674994 20087 catalog_manager.cc:4841] TS 1b64c563f60d4fb48895cbaaad57a82d (127.4.151.131:42699): tablet c59b15be1be14ed1a320fe09ad43f6bc (table test-table [id=7c3edb96e70048308955e8e84952fd31]) successfully deleted
I20260619 08:21:53.675530 20604 ts_tablet_manager.cc:1918] T c59b15be1be14ed1a320fe09ad43f6bc P aa44be97d3d24fc2b1ff2ea93acf6f9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:21:53.675595 20604 log.cc:1196] T c59b15be1be14ed1a320fe09ad43f6bc P aa44be97d3d24fc2b1ff2ea93acf6f9a: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/c59b15be1be14ed1a320fe09ad43f6bc
I20260619 08:21:53.675861 20604 ts_tablet_manager.cc:1939] T c59b15be1be14ed1a320fe09ad43f6bc P aa44be97d3d24fc2b1ff2ea93acf6f9a: Deleting consensus metadata
I20260619 08:21:53.676576 20090 catalog_manager.cc:4841] TS aa44be97d3d24fc2b1ff2ea93acf6f9a (127.4.151.130:42405): tablet c59b15be1be14ed1a320fe09ad43f6bc (table test-table [id=7c3edb96e70048308955e8e84952fd31]) successfully deleted
W20260619 08:21:53.683516 20087 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.151.129:32805 (error 108)
I20260619 08:21:53.683745 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 20299
I20260619 08:21:53.684552 20573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:21:53.684406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:32805 (local address 127.0.0.1:40004)
0619 08:21:53.684456 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 08:21:53.684507 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.129:32805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 08:21:53.685226 20087 catalog_manager.cc:4584] TS 0d8f97fb52e048f79923c60a7e51c4cb (127.4.151.129:32805): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c59b15be1be14ed1a320fe09ad43f6bc: Network error: Client connection negotiation failed: client connection to 127.4.151.129:32805: connect: Connection refused (error 111)
I20260619 08:21:53.688387 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 20433
I20260619 08:21:53.693369 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 20071
I20260619 08:22:13.699529 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:42845
--webserver_interface=127.4.151.190
--webserver_port=40417
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:42845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:22:13.781814 20606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:13.782089 20606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:13.782130 20606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:13.783371 20606 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 08:22:13.783442 20606 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 08:22:13.783466 20606 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 08:22:13.783485 20606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:22:13.783519 20606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:13.783545 20606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:13.783572 20606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:22:13.783602 20606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:22:13.785177 20606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:42845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:42845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=40417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:13.785507 20606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:13.785755 20606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:13.796854 20613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:13.796854 20612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:13.796957 20615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:13.796907 20606 server_base.cc:1034] running on GCE node
I20260619 08:22:13.797552 20606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:13.797909 20606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:13.799106 20606 hybrid_clock.cc:648] HybridClock initialized: now 1781857333799092 us; error 44 us; skew 500 ppm
Jun 19 08:22:13 dist-test-slave-d3n8 krb5kdc[20058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857333, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:22:13.802037 20606 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:22:13.802558 20606 webserver.cc:460] Webserver started at http://127.4.151.190:40417/ using document root <none> and password file <none>
I20260619 08:22:13.802834 20606 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:13.802892 20606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:13.804374 20606 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 08:22:13.805123 20622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:13.805419 20606 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:22:13.805521 20606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "ac68371830c44d3788301ebc2b4e80ef"
format_stamp: "Formatted at 2026-06-19 08:21:52 on dist-test-slave-d3n8"
server_key: "ebe58aebf5b36e0bc728b142c3cb4d1f"
server_key_iv: "4e22cec072027b152c5fb9bd462ab652"
server_key_version: "encryptionkey@0"
I20260619 08:22:13.805981 20606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:13.840210 20606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:13.842711 20606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:13.842854 20606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:13.847045 20606 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:42845
I20260619 08:22:13.847105 20674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:42845 every 8 connection(s)
I20260619 08:22:13.847476 20606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:22:13.849067 20675 sys_catalog.cc:263] Verifying existing consensus state
I20260619 08:22:13.849862 20675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef: Bootstrap starting.
I20260619 08:22:13.855410 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 20606
I20260619 08:22:13.855795 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:32805
--local_ip_for_outbound_sockets=127.4.151.129
--tserver_master_addrs=127.4.151.190:42845
--webserver_port=41251
--webserver_interface=127.4.151.129
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:22:13.858503 20675 log.cc:824] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:13.860558 20675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:22:13.860883 20675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef: Bootstrap complete.
I20260619 08:22:13.863315 20675 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac68371830c44d3788301ebc2b4e80ef" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42845 } }
I20260619 08:22:13.863574 20675 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac68371830c44d3788301ebc2b4e80ef, State: Initialized, Role: FOLLOWER
I20260619 08:22:13.863760 20675 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac68371830c44d3788301ebc2b4e80ef" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42845 } }
I20260619 08:22:13.863838 20675 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:22:13.863895 20675 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:22:13.863942 20675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [term 1 FOLLOWER]: Advancing to term 2
I20260619 08:22:13.864987 20675 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac68371830c44d3788301ebc2b4e80ef" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42845 } }
I20260619 08:22:13.865120 20675 leader_election.cc:302] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac68371830c44d3788301ebc2b4e80ef; no voters:
I20260619 08:22:13.865342 20675 leader_election.cc:288] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 08:22:13.865396 20679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [term 2 FOLLOWER]: Leader election won for term 2
I20260619 08:22:13.865520 20679 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [term 2 LEADER]: Becoming Leader. State: Replica: ac68371830c44d3788301ebc2b4e80ef, State: Running, Role: LEADER
I20260619 08:22:13.865636 20675 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:22:13.865679 20679 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac68371830c44d3788301ebc2b4e80ef" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42845 } }
I20260619 08:22:13.866062 20681 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac68371830c44d3788301ebc2b4e80ef. Latest consensus state: current_term: 2 leader_uuid: "ac68371830c44d3788301ebc2b4e80ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac68371830c44d3788301ebc2b4e80ef" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42845 } } }
I20260619 08:22:13.866076 20680 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ac68371830c44d3788301ebc2b4e80ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac68371830c44d3788301ebc2b4e80ef" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42845 } } }
I20260619 08:22:13.866158 20681 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [sys.catalog]: This master's current role is: LEADER
I20260619 08:22:13.866182 20680 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [sys.catalog]: This master's current role is: LEADER
I20260619 08:22:13.866391 20686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:22:13.867302 20686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:22:13.867514 20686 catalog_manager.cc:1259] Loaded cluster ID: e57fa568d99342e880ca166e9c4ded89
I20260619 08:22:13.867690 20686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:22:13.868846 20686 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:22:13.869024 20686 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef: Loaded TSK: 0
I20260619 08:22:13.879253 20686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef: Generated new TSK 1
I20260619 08:22:13.879361 20686 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 08:22:13.941987 20677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:13.942332 20677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:13.942444 20677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:13.943851 20677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:13.943912 20677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:13.943948 20677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:22:13.945811 20677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:32805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=41251
--tserver_master_addrs=127.4.151.190:42845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:13.946072 20677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:13.946337 20677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:13.952381 20703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:13.952479 20705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:13.952373 20702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:13.952800 20677 server_base.cc:1034] running on GCE node
I20260619 08:22:13.953043 20677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:13.953306 20677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:13.954479 20677 hybrid_clock.cc:648] HybridClock initialized: now 1781857333954458 us; error 41 us; skew 500 ppm
Jun 19 08:22:13 dist-test-slave-d3n8 krb5kdc[20058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857333, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:22:13.957041 20677 init.cc:377] Logged in from keytab as kudu/127.4.151.129@KRBTEST.COM (short username kudu)
I20260619 08:22:13.957613 20677 webserver.cc:460] Webserver started at http://127.4.151.129:41251/ using document root <none> and password file <none>
I20260619 08:22:13.957878 20677 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:13.957978 20677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:13.959435 20677 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:22:13.960233 20712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:13.960490 20677 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:22:13.960578 20677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "0d8f97fb52e048f79923c60a7e51c4cb"
format_stamp: "Formatted at 2026-06-19 08:21:52 on dist-test-slave-d3n8"
server_key: "03a8d733d06804c0d907afe8b26a8f1e"
server_key_iv: "eeda87f036f419bdeffc665a1c545bf1"
server_key_version: "encryptionkey@0"
I20260619 08:22:13.961061 20677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:13.972985 20677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:13.975733 20677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:13.975915 20677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:13.976210 20677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:22:13.976760 20719 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 08:22:13.978366 20677 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 08:22:13.978428 20677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 08:22:13.978464 20677 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 08:22:13.979305 20677 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 08:22:13.979351 20677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 08:22:13.979452 20719 tablet_bootstrap.cc:492] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb: Bootstrap starting.
I20260619 08:22:13.986658 20677 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:32805
I20260619 08:22:13.986814 20826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:32805 every 8 connection(s)
I20260619 08:22:13.987183 20677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
I20260619 08:22:13.990689 20719 log.cc:824] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:13.991133 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 20677
I20260619 08:22:13.991418 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:42405
--local_ip_for_outbound_sockets=127.4.151.130
--tserver_master_addrs=127.4.151.190:42845
--webserver_port=44915
--webserver_interface=127.4.151.130
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:22:13.994606 20719 tablet_bootstrap.cc:492] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:22:13.995357 20719 tablet_bootstrap.cc:492] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb: Bootstrap complete.
Jun 19 08:22:13 dist-test-slave-d3n8 krb5kdc[20058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857333, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:22:13.996711 20719 ts_tablet_manager.cc:1397] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.006s
I20260619 08:22:13.998818 20719 raft_consensus.cc:348] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } }
I20260619 08:22:13.999251 20719 raft_consensus.cc:729] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d8f97fb52e048f79923c60a7e51c4cb, State: Initialized, Role: FOLLOWER
I20260619 08:22:13.999475 20719 consensus_queue.cc:260] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } }
I20260619 08:22:13.999833 20719 ts_tablet_manager.cc:1428] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260619 08:22:14.001945 20830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:13.988071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:45169 (local address 127.4.151.190:42845)
0619 08:22:13.988312 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:13.988323 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:13.988371 (+ 48us) server_negotiation.cc:407] Connection header received
0619 08:22:13.989160 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:13.989168 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:13.989233 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:13.989430 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:13.990309 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:13.991814 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:13.993484 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:13.994078 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:13.994179 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:13.996970 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:13.996974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:13.996974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:22:13.996986 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:13.999578 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:14.000289 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:14.000293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:14.000294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:14.000456 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:14.000787 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:14.000787 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:14.000788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:14.001078 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:14.001165 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:14.001582 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 08:22:14.001728 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":81,"threads_started":1}
I20260619 08:22:14.001972 20829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:13.987985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.129:45169)
0619 08:22:13.988195 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:13.988245 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:13.988882 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:13.989482 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:13.989487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:13.989884 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:22:13.990159 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:13.990166 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:13.991982 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:13.991984 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:13.993234 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:13.993238 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:13.994291 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:13.994293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:13.994429 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:13.995061 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:22:13.995071 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:22:13.996850 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:22:13.999738 (+ 2888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:13.999745 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:13.999747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:14.000120 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:14.000535 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:14.000536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:14.000537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:14.000699 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:14.001169 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:22:14.001176 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:22:14.001364 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 08:22:14.001519 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 08:22:14.001719 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":99,"threads_started":1}
I20260619 08:22:14.003218 20827 heartbeater.cc:347] Connected to a master server at 127.4.151.190:42845
I20260619 08:22:14.003329 20827 heartbeater.cc:464] Registering TS with master...
I20260619 08:22:14.003526 20827 heartbeater.cc:511] Master 127.4.151.190:42845 requested a full tablet report, sending...
I20260619 08:22:14.004330 20639 ts_manager.cc:194] Registered new tserver with Master: 0d8f97fb52e048f79923c60a7e51c4cb (127.4.151.129:32805)
I20260619 08:22:14.005534 20639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.129@KRBTEST.COM'} at 127.4.151.129:45169
I20260619 08:22:14.006669 20827 heartbeater.cc:503] Master 127.4.151.190:42845 was elected leader, sending a full tablet report...
I20260619 08:22:14.013862 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:14.005273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44820 (local address 127.4.151.129:32805)
0619 08:22:14.009711 (+ 4438us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:14.009716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:14.009725 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:22:14.009758 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:14.009761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:14.009775 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:14.009898 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:14.010074 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.011255 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.012624 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.013344 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.013484 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:14.013546 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:14.013745 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 08:22:14.013783 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4404,"thread_start_us":76,"threads_started":1}
I20260619 08:22:14.013887 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:14.004992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:32805 (local address 127.0.0.1:44820)
0619 08:22:14.009477 (+ 4485us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:14.009516 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:14.009584 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:14.009920 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:14.009923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:14.009926 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:14.009995 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:14.009997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.011339 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.011340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:14.012530 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:14.012534 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.013508 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.013510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:14.013634 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:14.013651 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 08:22:14.013725 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 08:22:14.013816 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4452,"thread_start_us":96,"threads_started":1}
I20260619 08:22:14.014614 20744 tablet_service.cc:1514] Processing DeleteTablet for tablet c59b15be1be14ed1a320fe09ad43f6bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:21:53 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:44820
I20260619 08:22:14.015295 20836 tablet_replica.cc:330] stopping tablet replica
I20260619 08:22:14.015413 20836 raft_consensus.cc:2229] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:22:14.015486 20836 raft_consensus.cc:2258] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:22:14.015802 20836 ts_tablet_manager.cc:1905] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:22:14.018057 20836 ts_tablet_manager.cc:1918] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:22:14.018293 20836 log.cc:1196] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/wal/wals/c59b15be1be14ed1a320fe09ad43f6bc
I20260619 08:22:14.018564 20836 ts_tablet_manager.cc:1939] T c59b15be1be14ed1a320fe09ad43f6bc P 0d8f97fb52e048f79923c60a7e51c4cb: Deleting consensus metadata
I20260619 08:22:14.019081 20623 catalog_manager.cc:4841] TS 0d8f97fb52e048f79923c60a7e51c4cb (127.4.151.129:32805): tablet c59b15be1be14ed1a320fe09ad43f6bc (table test-table [id=7c3edb96e70048308955e8e84952fd31]) successfully deleted
W20260619 08:22:14.082145 20832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:14.082396 20832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:14.082432 20832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:14.083607 20832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:14.083683 20832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:14.083740 20832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:22:14.085251 20832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:42405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=44915
--tserver_master_addrs=127.4.151.190:42845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:14.085551 20832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:14.085790 20832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:14.091107 20844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:14.091178 20842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:14.091107 20841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:14.091408 20832 server_base.cc:1034] running on GCE node
I20260619 08:22:14.091646 20832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:14.091845 20832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:14.093003 20832 hybrid_clock.cc:648] HybridClock initialized: now 1781857334092982 us; error 27 us; skew 500 ppm
Jun 19 08:22:14 dist-test-slave-d3n8 krb5kdc[20058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857334, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:22:14.095832 20832 init.cc:377] Logged in from keytab as kudu/127.4.151.130@KRBTEST.COM (short username kudu)
I20260619 08:22:14.096311 20832 webserver.cc:460] Webserver started at http://127.4.151.130:44915/ using document root <none> and password file <none>
I20260619 08:22:14.096544 20832 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:14.096601 20832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:14.098158 20832 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260619 08:22:14.098822 20851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:14.099053 20832 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:22:14.099148 20832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a"
format_stamp: "Formatted at 2026-06-19 08:21:53 on dist-test-slave-d3n8"
server_key: "57bdb51ffc163a93e812d327133fa0cd"
server_key_iv: "0fbd96f440d5fa646955ee46d157b44f"
server_key_version: "encryptionkey@0"
I20260619 08:22:14.099547 20832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:14.115799 20832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:14.117966 20832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:14.118113 20832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:14.118364 20832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:22:14.118677 20832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:22:14.118721 20832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:14.118744 20832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:22:14.118757 20832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:14.124897 20832 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:42405
I20260619 08:22:14.124953 20964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:42405 every 8 connection(s)
I20260619 08:22:14.125272 20832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
I20260619 08:22:14.127632 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 20832
I20260619 08:22:14.127902 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:42699
--local_ip_for_outbound_sockets=127.4.151.131
--tserver_master_addrs=127.4.151.190:42845
--webserver_port=40871
--webserver_interface=127.4.151.131
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 08:22:14 dist-test-slave-d3n8 krb5kdc[20058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857334, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:22:14.135339 20830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:14.126179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:60141 (local address 127.4.151.190:42845)
0619 08:22:14.126296 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:14.126298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:14.126307 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:22:14.126849 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:14.126851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:14.126866 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:14.126945 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:14.128076 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.128639 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.129888 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.130250 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.130311 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:14.132472 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:14.132473 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:14.132474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:22:14.132482 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:14.133924 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:14.134358 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:14.134359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:14.134360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:14.134418 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:14.134673 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:14.134674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:14.134674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:14.134788 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:14.134863 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:14.135134 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 08:22:14.135243 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 08:22:14.135497 20967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:14.125925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.130:60141)
0619 08:22:14.126146 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:14.126193 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:14.126736 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:14.127013 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:14.127017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:14.127643 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:22:14.127914 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:14.127919 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.128742 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.128743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:14.129792 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:14.129795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.130329 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.130330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:14.130431 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:14.130967 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:22:14.130975 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:22:14.132353 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:22:14.134021 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:14.134027 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:14.134042 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:14.134268 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:14.134487 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:14.134487 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:14.134488 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:14.134598 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:14.134854 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:22:14.134867 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:22:14.135002 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 08:22:14.135114 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 08:22:14.135269 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":71,"threads_started":1}
I20260619 08:22:14.136101 20965 heartbeater.cc:347] Connected to a master server at 127.4.151.190:42845
I20260619 08:22:14.136216 20965 heartbeater.cc:464] Registering TS with master...
I20260619 08:22:14.136422 20965 heartbeater.cc:511] Master 127.4.151.190:42845 requested a full tablet report, sending...
I20260619 08:22:14.136863 20639 ts_manager.cc:194] Registered new tserver with Master: aa44be97d3d24fc2b1ff2ea93acf6f9a (127.4.151.130:42405)
I20260619 08:22:14.137292 20639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.130@KRBTEST.COM'} at 127.4.151.130:60141
W20260619 08:22:14.211570 20968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:14.211783 20968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:14.211807 20968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:14.213037 20968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:14.213083 20968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:14.213111 20968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:22:14.214787 20968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:42699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=40871
--tserver_master_addrs=127.4.151.190:42845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:14.215188 20968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:14.215436 20968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:14.220613 20973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:14.220620 20974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:14.220778 20968 server_base.cc:1034] running on GCE node
W20260619 08:22:14.220642 20976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:14.221082 20968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:14.221323 20968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:14.222488 20968 hybrid_clock.cc:648] HybridClock initialized: now 1781857334222472 us; error 32 us; skew 500 ppm
Jun 19 08:22:14 dist-test-slave-d3n8 krb5kdc[20058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857334, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:22:14.225358 20968 init.cc:377] Logged in from keytab as kudu/127.4.151.131@KRBTEST.COM (short username kudu)
I20260619 08:22:14.225884 20968 webserver.cc:460] Webserver started at http://127.4.151.131:40871/ using document root <none> and password file <none>
I20260619 08:22:14.226158 20968 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:14.226222 20968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:14.227698 20968 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 08:22:14.228605 20983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:14.228801 20968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:22:14.228917 20968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "1b64c563f60d4fb48895cbaaad57a82d"
format_stamp: "Formatted at 2026-06-19 08:21:53 on dist-test-slave-d3n8"
server_key: "f208fd339cf2bd3bb06eafea1ac11054"
server_key_iv: "eb4523c764e3871f9845a418035dba63"
server_key_version: "encryptionkey@0"
I20260619 08:22:14.229423 20968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:14.247637 20968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:14.250322 20968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:14.250509 20968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:14.250774 20968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:22:14.251127 20968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:22:14.251195 20968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:14.251242 20968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:22:14.251274 20968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:14.257824 20968 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:42699
I20260619 08:22:14.257870 21096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:42699 every 8 connection(s)
I20260619 08:22:14.258255 20968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
I20260619 08:22:14.263497 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 20968
Jun 19 08:22:14 dist-test-slave-d3n8 krb5kdc[20058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857334, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:22:14.269038 20830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:14.259240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:57617 (local address 127.4.151.190:42845)
0619 08:22:14.259297 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:14.259299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:14.259424 (+ 125us) server_negotiation.cc:407] Connection header received
0619 08:22:14.259942 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:14.259942 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:14.259953 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:14.260005 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:14.260995 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.261535 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.262584 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.262775 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.262830 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:14.265476 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:14.265477 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:14.265482 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:22:14.265490 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:14.267354 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:14.267863 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:14.267864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:14.267864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:14.267931 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:14.268332 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:14.268333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:14.268333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:14.268455 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:14.268543 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:14.268839 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 08:22:14.268935 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 08:22:14.269294 21099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:14.259085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.131:57617)
0619 08:22:14.259274 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:14.259324 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:14.259844 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:14.260070 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:14.260073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:14.260681 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:22:14.260888 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:14.260893 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.261654 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.261655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:14.262504 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:14.262506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.262850 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.262851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:14.262953 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:14.263584 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:22:14.263592 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:22:14.265338 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:22:14.267432 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:14.267438 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:14.267440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:14.267787 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:14.268035 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:14.268036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:14.268036 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:14.268235 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:14.268533 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:22:14.268540 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:22:14.268695 (+ 155us) client_negotiation.cc:769] Sending connection context
0619 08:22:14.268816 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 08:22:14.268996 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":74,"threads_started":1}
I20260619 08:22:14.269848 21097 heartbeater.cc:347] Connected to a master server at 127.4.151.190:42845
I20260619 08:22:14.269965 21097 heartbeater.cc:464] Registering TS with master...
I20260619 08:22:14.270172 21097 heartbeater.cc:511] Master 127.4.151.190:42845 requested a full tablet report, sending...
I20260619 08:22:14.270602 20639 ts_manager.cc:194] Registered new tserver with Master: 1b64c563f60d4fb48895cbaaad57a82d (127.4.151.131:42699)
I20260619 08:22:14.270995 20639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.131@KRBTEST.COM'} at 127.4.151.131:57617
I20260619 08:22:14.272004 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:14.263954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51284 (local address 127.4.151.190:42845)
0619 08:22:14.264241 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:14.264260 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:14.264268 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:22:14.264291 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:14.264292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:14.264304 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:14.264369 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:14.265096 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.265897 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.267331 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.267606 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.267685 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:14.268897 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:14.268898 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:14.268899 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:22:14.268905 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:14.270835 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:14.271181 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:14.271181 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:14.271182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:14.271249 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:14.271479 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:14.271480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:14.271480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:14.271599 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:14.271772 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:14.271810 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 08:22:14.271888 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":75,"threads_started":1}
I20260619 08:22:14.272779 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:22:14.275458 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:14.273087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51290 (local address 127.4.151.190:42845)
0619 08:22:14.273128 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:14.273129 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:14.273238 (+ 109us) server_negotiation.cc:407] Connection header received
0619 08:22:14.273322 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:14.273322 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:14.273349 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:14.273454 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:22:14.273566 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.274001 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.274808 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.275022 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.275077 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:14.275252 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:22:14.275299 (+ 47us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 08:22:14.275417 (+ 118us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51290: token expired
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 08:22:14.275498 21101 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51290: token expired
W20260619 08:22:14.275661 20568 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.151.190:42845)
I20260619 08:22:14.281354 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:14.275926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51296 (local address 127.4.151.190:42845)
0619 08:22:14.275958 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:14.275960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:14.275967 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:22:14.276016 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:14.276017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:14.276023 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:14.276067 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:14.276503 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.276941 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.277809 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.277993 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.278059 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:14.279132 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:14.279133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:14.279133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:22:14.279139 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:14.280401 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:14.280727 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:14.280727 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:14.280728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:14.280779 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:14.280956 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:14.280957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:14.280957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:14.281072 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:14.281163 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:14.281243 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 08:22:14.281300 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260619 08:22:14.282562 20639 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:22:14.284155 20639 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:22:14.286415 20744 tablet_service.cc:1467] Processing CreateTablet for tablet ae1e202809864c0facdbf51e60e2e3b5 (DEFAULT_TABLE table=test-table [id=b985b812b7a34b13807765e7e168f067]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:22:14.286621 20744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1e202809864c0facdbf51e60e2e3b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:14.288192 20719 tablet_bootstrap.cc:492] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb: Bootstrap starting.
I20260619 08:22:14.288545 20719 tablet_bootstrap.cc:654] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:14.289126 20719 tablet_bootstrap.cc:492] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb: No bootstrap required, opened a new log
I20260619 08:22:14.289204 20719 ts_tablet_manager.cc:1397] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260619 08:22:14.289315 20719 raft_consensus.cc:348] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } }
I20260619 08:22:14.289412 20719 raft_consensus.cc:374] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:14.289463 20719 raft_consensus.cc:729] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d8f97fb52e048f79923c60a7e51c4cb, State: Initialized, Role: FOLLOWER
I20260619 08:22:14.289505 20719 consensus_queue.cc:260] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } }
I20260619 08:22:14.289597 20719 ts_tablet_manager.cc:1428] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:14.290365 21103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:14.286135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:42405 (local address 127.0.0.1:56972)
0619 08:22:14.286358 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:14.286368 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:14.286417 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:14.286770 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:14.286771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:14.286774 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:14.286830 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:14.286832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.287804 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.287805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:14.289097 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:14.289100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.290117 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.290118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:14.290184 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:14.290188 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:22:14.290248 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 08:22:14.290294 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":110,"threads_started":1}
I20260619 08:22:14.290393 21105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:14.286438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.4.151.130:42405)
0619 08:22:14.286616 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:14.286621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:14.286631 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:22:14.286678 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:14.286679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:14.286692 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:14.286759 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:14.286899 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.287717 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.289191 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.289994 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.290069 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:14.290128 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:14.290275 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 08:22:14.290318 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":109,"threads_started":1}
I20260619 08:22:14.291114 20899 tablet_service.cc:1467] Processing CreateTablet for tablet ae1e202809864c0facdbf51e60e2e3b5 (DEFAULT_TABLE table=test-table [id=b985b812b7a34b13807765e7e168f067]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:22:14.291460 20899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1e202809864c0facdbf51e60e2e3b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:14.292110 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:14.286074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:42699 (local address 127.0.0.1:40538)
0619 08:22:14.286136 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:14.286149 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:14.286272 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:14.288436 (+ 2164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:14.288436 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:14.288438 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:14.288479 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:14.288480 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.289843 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.289844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:14.290987 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:14.290989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.291911 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.291912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:14.291962 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:14.291965 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:22:14.292020 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 08:22:14.292056 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 08:22:14.292186 21104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:14.286169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40538 (local address 127.4.151.131:42699)
0619 08:22:14.288231 (+ 2062us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:14.288246 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:14.288258 (+ 12us) server_negotiation.cc:407] Connection header received
0619 08:22:14.288299 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:14.288301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:14.288318 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:14.288428 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:14.288555 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.289782 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.291064 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.291782 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.291864 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:14.291922 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:14.292052 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 08:22:14.292100 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2024,"thread_start_us":92,"threads_started":1}
I20260619 08:22:14.294483 21107 tablet_bootstrap.cc:492] T ae1e202809864c0facdbf51e60e2e3b5 P aa44be97d3d24fc2b1ff2ea93acf6f9a: Bootstrap starting.
I20260619 08:22:14.294986 21031 tablet_service.cc:1467] Processing CreateTablet for tablet ae1e202809864c0facdbf51e60e2e3b5 (DEFAULT_TABLE table=test-table [id=b985b812b7a34b13807765e7e168f067]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:22:14.295217 21107 tablet_bootstrap.cc:654] T ae1e202809864c0facdbf51e60e2e3b5 P aa44be97d3d24fc2b1ff2ea93acf6f9a: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:14.295315 21031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1e202809864c0facdbf51e60e2e3b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:14.295753 21107 log.cc:824] T ae1e202809864c0facdbf51e60e2e3b5 P aa44be97d3d24fc2b1ff2ea93acf6f9a: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:14.296698 21107 tablet_bootstrap.cc:492] T ae1e202809864c0facdbf51e60e2e3b5 P aa44be97d3d24fc2b1ff2ea93acf6f9a: No bootstrap required, opened a new log
I20260619 08:22:14.296784 21107 ts_tablet_manager.cc:1397] T ae1e202809864c0facdbf51e60e2e3b5 P aa44be97d3d24fc2b1ff2ea93acf6f9a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:22:14.298357 21109 tablet_bootstrap.cc:492] T ae1e202809864c0facdbf51e60e2e3b5 P 1b64c563f60d4fb48895cbaaad57a82d: Bootstrap starting.
I20260619 08:22:14.298240 21107 raft_consensus.cc:348] T ae1e202809864c0facdbf51e60e2e3b5 P aa44be97d3d24fc2b1ff2ea93acf6f9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } }
I20260619 08:22:14.298434 21107 raft_consensus.cc:374] T ae1e202809864c0facdbf51e60e2e3b5 P aa44be97d3d24fc2b1ff2ea93acf6f9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:14.298492 21107 raft_consensus.cc:729] T ae1e202809864c0facdbf51e60e2e3b5 P aa44be97d3d24fc2b1ff2ea93acf6f9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa44be97d3d24fc2b1ff2ea93acf6f9a, State: Initialized, Role: FOLLOWER
I20260619 08:22:14.298652 21107 consensus_queue.cc:260] T ae1e202809864c0facdbf51e60e2e3b5 P aa44be97d3d24fc2b1ff2ea93acf6f9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } }
I20260619 08:22:14.298960 21107 ts_tablet_manager.cc:1428] T ae1e202809864c0facdbf51e60e2e3b5 P aa44be97d3d24fc2b1ff2ea93acf6f9a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:22:14.298987 20965 heartbeater.cc:503] Master 127.4.151.190:42845 was elected leader, sending a full tablet report...
I20260619 08:22:14.299137 21109 tablet_bootstrap.cc:654] T ae1e202809864c0facdbf51e60e2e3b5 P 1b64c563f60d4fb48895cbaaad57a82d: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:14.299712 21109 log.cc:824] T ae1e202809864c0facdbf51e60e2e3b5 P 1b64c563f60d4fb48895cbaaad57a82d: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:14.300597 21109 tablet_bootstrap.cc:492] T ae1e202809864c0facdbf51e60e2e3b5 P 1b64c563f60d4fb48895cbaaad57a82d: No bootstrap required, opened a new log
I20260619 08:22:14.300657 21109 ts_tablet_manager.cc:1397] T ae1e202809864c0facdbf51e60e2e3b5 P 1b64c563f60d4fb48895cbaaad57a82d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:22:14.302241 21109 raft_consensus.cc:348] T ae1e202809864c0facdbf51e60e2e3b5 P 1b64c563f60d4fb48895cbaaad57a82d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } }
I20260619 08:22:14.302410 21109 raft_consensus.cc:374] T ae1e202809864c0facdbf51e60e2e3b5 P 1b64c563f60d4fb48895cbaaad57a82d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:14.302434 21109 raft_consensus.cc:729] T ae1e202809864c0facdbf51e60e2e3b5 P 1b64c563f60d4fb48895cbaaad57a82d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b64c563f60d4fb48895cbaaad57a82d, State: Initialized, Role: FOLLOWER
I20260619 08:22:14.302558 21109 consensus_queue.cc:260] T ae1e202809864c0facdbf51e60e2e3b5 P 1b64c563f60d4fb48895cbaaad57a82d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } }
I20260619 08:22:14.302798 21109 ts_tablet_manager.cc:1428] T ae1e202809864c0facdbf51e60e2e3b5 P 1b64c563f60d4fb48895cbaaad57a82d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:22:14.302860 21097 heartbeater.cc:503] Master 127.4.151.190:42845 was elected leader, sending a full tablet report...
W20260619 08:22:14.375831 20966 tablet.cc:2369] T ae1e202809864c0facdbf51e60e2e3b5 P aa44be97d3d24fc2b1ff2ea93acf6f9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:22:14.455077 20833 raft_consensus.cc:482] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:22:14.455219 20833 raft_consensus.cc:504] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } }
I20260619 08:22:14.455554 20833 leader_election.cc:288] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b64c563f60d4fb48895cbaaad57a82d (127.4.151.131:42699), aa44be97d3d24fc2b1ff2ea93acf6f9a (127.4.151.130:42405)
I20260619 08:22:14.459525 21105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:14.455926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:37173 (local address 127.4.151.130:42405)
0619 08:22:14.455975 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:14.455977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:14.456136 (+ 159us) server_negotiation.cc:407] Connection header received
0619 08:22:14.456161 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:14.456162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:14.456169 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:14.456230 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:14.456396 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.457089 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.458526 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.459156 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.459220 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:14.459226 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:14.459434 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 08:22:14.459469 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 08:22:14.459513 21104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:14.455850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:51447 (local address 127.4.151.131:42699)
0619 08:22:14.455927 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:14.455929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:14.455940 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:22:14.455973 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:14.455974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:14.455985 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:14.456051 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:14.456262 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.457035 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.458526 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.459101 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.459183 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:14.459196 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:14.459397 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 08:22:14.459443 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 08:22:14.459515 20829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:14.455783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:42699 (local address 127.4.151.129:51447)
0619 08:22:14.455848 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:14.455861 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:14.455909 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:14.456084 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:14.456085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:14.456088 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:14.456177 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:14.456179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.457152 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.457153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:14.458416 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:14.458419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.459190 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.459191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:14.459297 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:14.459301 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:22:14.459363 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 08:22:14.459417 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":9}
I20260619 08:22:14.459626 21113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:14.455801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:42405 (local address 127.4.151.129:37173)
0619 08:22:14.456011 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:14.456019 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:14.456094 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:14.456254 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:14.456254 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:14.456256 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:14.456327 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:14.456329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.457167 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.457167 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:14.458416 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:14.458419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.459268 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.459270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:14.459328 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:14.459331 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:22:14.459423 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 08:22:14.459455 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":5,"thread_start_us":101,"threads_started":1}
I20260619 08:22:14.459919 21051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1e202809864c0facdbf51e60e2e3b5" candidate_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b64c563f60d4fb48895cbaaad57a82d" is_pre_election: true
I20260619 08:22:14.459942 20919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1e202809864c0facdbf51e60e2e3b5" candidate_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" is_pre_election: true
I20260619 08:22:14.460094 21051 raft_consensus.cc:2454] T ae1e202809864c0facdbf51e60e2e3b5 P 1b64c563f60d4fb48895cbaaad57a82d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d8f97fb52e048f79923c60a7e51c4cb in term 0.
I20260619 08:22:14.460147 20919 raft_consensus.cc:2454] T ae1e202809864c0facdbf51e60e2e3b5 P aa44be97d3d24fc2b1ff2ea93acf6f9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d8f97fb52e048f79923c60a7e51c4cb in term 0.
I20260619 08:22:14.460330 20713 leader_election.cc:302] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d8f97fb52e048f79923c60a7e51c4cb, 1b64c563f60d4fb48895cbaaad57a82d; no voters:
I20260619 08:22:14.460486 20833 raft_consensus.cc:2781] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:22:14.460546 20833 raft_consensus.cc:482] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:22:14.460575 20833 raft_consensus.cc:3037] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:14.461306 20833 raft_consensus.cc:504] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } }
I20260619 08:22:14.461493 20833 leader_election.cc:288] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [CANDIDATE]: Term 1 election: Requested vote from peers 1b64c563f60d4fb48895cbaaad57a82d (127.4.151.131:42699), aa44be97d3d24fc2b1ff2ea93acf6f9a (127.4.151.130:42405)
I20260619 08:22:14.461625 21051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1e202809864c0facdbf51e60e2e3b5" candidate_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b64c563f60d4fb48895cbaaad57a82d"
I20260619 08:22:14.461704 21051 raft_consensus.cc:3037] T ae1e202809864c0facdbf51e60e2e3b5 P 1b64c563f60d4fb48895cbaaad57a82d [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:14.461692 20919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1e202809864c0facdbf51e60e2e3b5" candidate_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a"
I20260619 08:22:14.461763 20919 raft_consensus.cc:3037] T ae1e202809864c0facdbf51e60e2e3b5 P aa44be97d3d24fc2b1ff2ea93acf6f9a [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:14.462342 21051 raft_consensus.cc:2454] T ae1e202809864c0facdbf51e60e2e3b5 P 1b64c563f60d4fb48895cbaaad57a82d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d8f97fb52e048f79923c60a7e51c4cb in term 1.
I20260619 08:22:14.462371 20919 raft_consensus.cc:2454] T ae1e202809864c0facdbf51e60e2e3b5 P aa44be97d3d24fc2b1ff2ea93acf6f9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d8f97fb52e048f79923c60a7e51c4cb in term 1.
I20260619 08:22:14.462497 20713 leader_election.cc:302] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d8f97fb52e048f79923c60a7e51c4cb, 1b64c563f60d4fb48895cbaaad57a82d; no voters:
I20260619 08:22:14.462608 20833 raft_consensus.cc:2781] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:22:14.462752 20833 raft_consensus.cc:686] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 1 LEADER]: Becoming Leader. State: Replica: 0d8f97fb52e048f79923c60a7e51c4cb, State: Running, Role: LEADER
I20260619 08:22:14.462842 20833 consensus_queue.cc:237] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } }
I20260619 08:22:14.463657 20638 catalog_manager.cc:5495] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d8f97fb52e048f79923c60a7e51c4cb (127.4.151.129). New cstate: current_term: 1 leader_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } health_report { overall_health: HEALTHY } } }
I20260619 08:22:14.473855 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:14.471330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44828 (local address 127.4.151.129:32805)
0619 08:22:14.471383 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:14.471385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:14.471393 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:22:14.471426 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:14.471427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:14.471444 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:14.471515 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:22:14.471628 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.472229 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.472920 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:14.473196 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:14.473275 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:14.473503 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:22:14.473608 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:22:14.473688 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:14.473769 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 08:22:14.473790 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:22:14.475486 20919 raft_consensus.cc:1261] T ae1e202809864c0facdbf51e60e2e3b5 P aa44be97d3d24fc2b1ff2ea93acf6f9a [term 1 FOLLOWER]: Refusing update from remote peer 0d8f97fb52e048f79923c60a7e51c4cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:22:14.475486 21051 raft_consensus.cc:1261] T ae1e202809864c0facdbf51e60e2e3b5 P 1b64c563f60d4fb48895cbaaad57a82d [term 1 FOLLOWER]: Refusing update from remote peer 0d8f97fb52e048f79923c60a7e51c4cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:22:14.475853 20833 consensus_queue.cc:1035] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:22:14.476015 21114 consensus_queue.cc:1035] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:22:14.479568 21119 mvcc.cc:205] Tried to move back new op lower bound from 7298487642009141248 to 7298487641960448000. Current Snapshot: MvccSnapshot[applied={T|T < 7298487642009141248 or (T in {7298487642009141248})}]
I20260619 08:22:14.480628 20638 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:22:14.480723 20638 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:22:14.481465 20638 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef: Sending DeleteTablet for 3 replicas of tablet ae1e202809864c0facdbf51e60e2e3b5
I20260619 08:22:14.481793 20744 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1e202809864c0facdbf51e60e2e3b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:22:14 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:44820
I20260619 08:22:14.481819 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 20677
I20260619 08:22:14.481854 20899 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1e202809864c0facdbf51e60e2e3b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:22:14 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:56972
I20260619 08:22:14.481899 20836 tablet_replica.cc:330] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb: stopping tablet replica
I20260619 08:22:14.481942 20836 raft_consensus.cc:2229] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:22:14.482023 20836 raft_consensus.cc:2258] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:22:14.482311 21031 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1e202809864c0facdbf51e60e2e3b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:22:14 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:40538
I20260619 08:22:14.482508 21126 tablet_replica.cc:330] T ae1e202809864c0facdbf51e60e2e3b5 P aa44be97d3d24fc2b1ff2ea93acf6f9a: stopping tablet replica
I20260619 08:22:14.482522 21127 tablet_replica.cc:330] T ae1e202809864c0facdbf51e60e2e3b5 P 1b64c563f60d4fb48895cbaaad57a82d: stopping tablet replica
I20260619 08:22:14.482617 21127 raft_consensus.cc:2229] T ae1e202809864c0facdbf51e60e2e3b5 P 1b64c563f60d4fb48895cbaaad57a82d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:22:14.482622 21126 raft_consensus.cc:2229] T ae1e202809864c0facdbf51e60e2e3b5 P aa44be97d3d24fc2b1ff2ea93acf6f9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:22:14.482723 21127 raft_consensus.cc:2258] T ae1e202809864c0facdbf51e60e2e3b5 P 1b64c563f60d4fb48895cbaaad57a82d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:22:14.482723 21126 raft_consensus.cc:2258] T ae1e202809864c0facdbf51e60e2e3b5 P aa44be97d3d24fc2b1ff2ea93acf6f9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:22:14.483119 21127 ts_tablet_manager.cc:1905] T ae1e202809864c0facdbf51e60e2e3b5 P 1b64c563f60d4fb48895cbaaad57a82d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:22:14.484593 21126 ts_tablet_manager.cc:1905] T ae1e202809864c0facdbf51e60e2e3b5 P aa44be97d3d24fc2b1ff2ea93acf6f9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:22:14.484601 21127 ts_tablet_manager.cc:1918] T ae1e202809864c0facdbf51e60e2e3b5 P 1b64c563f60d4fb48895cbaaad57a82d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:22:14.484673 21127 log.cc:1196] T ae1e202809864c0facdbf51e60e2e3b5 P 1b64c563f60d4fb48895cbaaad57a82d: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/ae1e202809864c0facdbf51e60e2e3b5
I20260619 08:22:14.484967 21127 ts_tablet_manager.cc:1939] T ae1e202809864c0facdbf51e60e2e3b5 P 1b64c563f60d4fb48895cbaaad57a82d: Deleting consensus metadata
I20260619 08:22:14.485414 20623 catalog_manager.cc:4841] TS 1b64c563f60d4fb48895cbaaad57a82d (127.4.151.131:42699): tablet ae1e202809864c0facdbf51e60e2e3b5 (table test-table [id=b985b812b7a34b13807765e7e168f067]) successfully deleted
I20260619 08:22:14.486173 21126 ts_tablet_manager.cc:1918] T ae1e202809864c0facdbf51e60e2e3b5 P aa44be97d3d24fc2b1ff2ea93acf6f9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:22:14.486231 21126 log.cc:1196] T ae1e202809864c0facdbf51e60e2e3b5 P aa44be97d3d24fc2b1ff2ea93acf6f9a: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/ae1e202809864c0facdbf51e60e2e3b5
I20260619 08:22:14.486502 21126 ts_tablet_manager.cc:1939] T ae1e202809864c0facdbf51e60e2e3b5 P aa44be97d3d24fc2b1ff2ea93acf6f9a: Deleting consensus metadata
I20260619 08:22:14.486912 20626 catalog_manager.cc:4841] TS aa44be97d3d24fc2b1ff2ea93acf6f9a (127.4.151.130:42405): tablet ae1e202809864c0facdbf51e60e2e3b5 (table test-table [id=b985b812b7a34b13807765e7e168f067]) successfully deleted
W20260619 08:22:14.488096 20623 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.151.129:32805 (error 108)
W20260619 08:22:14.488579 20834 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:22:14.488437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:32805 (local address 127.0.0.1:44842)
0619 08:22:14.488495 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:14.488543 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.129:32805: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 08:22:14.488744 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 20832
W20260619 08:22:14.488731 20623 catalog_manager.cc:4584] TS 0d8f97fb52e048f79923c60a7e51c4cb (127.4.151.129:32805): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae1e202809864c0facdbf51e60e2e3b5: Network error: Client connection negotiation failed: client connection to 127.4.151.129:32805: connect: Connection reset by peer (error 104)
I20260619 08:22:14.493330 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 20968
I20260619 08:22:14.497663 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 20606
I20260619 08:22:34.503127 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:42845
--webserver_interface=127.4.151.190
--webserver_port=40417
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:42845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:22:34.586714 21129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:34.586941 21129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:34.586963 21129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:34.588119 21129 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 08:22:34.588160 21129 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 08:22:34.588173 21129 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 08:22:34.588186 21129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:22:34.588198 21129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:34.588212 21129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:34.588223 21129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:22:34.588235 21129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:22:34.589726 21129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:42845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:42845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=40417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:34.589941 21129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:34.590117 21129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:34.596063 21134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:34.596086 21129 server_base.cc:1034] running on GCE node
W20260619 08:22:34.596249 21135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:34.596066 21137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:34.596585 21129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:34.596959 21129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:34.598331 21129 hybrid_clock.cc:648] HybridClock initialized: now 1781857354598360 us; error 102 us; skew 500 ppm
Jun 19 08:22:34 dist-test-slave-d3n8 krb5kdc[20058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857354, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:22:34.601046 21129 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:22:34.601663 21129 webserver.cc:460] Webserver started at http://127.4.151.190:40417/ using document root <none> and password file <none>
I20260619 08:22:34.601897 21129 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:34.601966 21129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:34.603443 21129 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:22:34.604332 21144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:34.604555 21129 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:22:34.604677 21129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "ac68371830c44d3788301ebc2b4e80ef"
format_stamp: "Formatted at 2026-06-19 08:21:52 on dist-test-slave-d3n8"
server_key: "ebe58aebf5b36e0bc728b142c3cb4d1f"
server_key_iv: "4e22cec072027b152c5fb9bd462ab652"
server_key_version: "encryptionkey@0"
I20260619 08:22:34.605136 21129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:34.635830 21129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:34.638123 21129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:34.638386 21129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:34.642446 21129 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:42845
I20260619 08:22:34.642503 21196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:42845 every 8 connection(s)
I20260619 08:22:34.642907 21129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:22:34.644485 21197 sys_catalog.cc:263] Verifying existing consensus state
I20260619 08:22:34.645200 21197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef: Bootstrap starting.
I20260619 08:22:34.649335 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 21129
I20260619 08:22:34.649713 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:32805
--local_ip_for_outbound_sockets=127.4.151.129
--tserver_master_addrs=127.4.151.190:42845
--webserver_port=41251
--webserver_interface=127.4.151.129
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:22:34.654215 21197 log.cc:824] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:34.656468 21197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:22:34.656805 21197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef: Bootstrap complete.
I20260619 08:22:34.659236 21197 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac68371830c44d3788301ebc2b4e80ef" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42845 } }
I20260619 08:22:34.659471 21197 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac68371830c44d3788301ebc2b4e80ef, State: Initialized, Role: FOLLOWER
I20260619 08:22:34.659634 21197 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac68371830c44d3788301ebc2b4e80ef" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42845 } }
I20260619 08:22:34.659711 21197 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:22:34.659777 21197 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:22:34.659818 21197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [term 2 FOLLOWER]: Advancing to term 3
I20260619 08:22:34.660841 21197 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac68371830c44d3788301ebc2b4e80ef" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42845 } }
I20260619 08:22:34.660953 21197 leader_election.cc:302] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac68371830c44d3788301ebc2b4e80ef; no voters:
I20260619 08:22:34.661144 21197 leader_election.cc:288] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 08:22:34.661180 21201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [term 3 FOLLOWER]: Leader election won for term 3
I20260619 08:22:34.661350 21201 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [term 3 LEADER]: Becoming Leader. State: Replica: ac68371830c44d3788301ebc2b4e80ef, State: Running, Role: LEADER
I20260619 08:22:34.661445 21197 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:22:34.661514 21201 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac68371830c44d3788301ebc2b4e80ef" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42845 } }
I20260619 08:22:34.661784 21201 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac68371830c44d3788301ebc2b4e80ef. Latest consensus state: current_term: 3 leader_uuid: "ac68371830c44d3788301ebc2b4e80ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac68371830c44d3788301ebc2b4e80ef" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42845 } } }
I20260619 08:22:34.661873 21201 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [sys.catalog]: This master's current role is: LEADER
I20260619 08:22:34.661791 21202 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ac68371830c44d3788301ebc2b4e80ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac68371830c44d3788301ebc2b4e80ef" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 42845 } } }
I20260619 08:22:34.662031 21202 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef [sys.catalog]: This master's current role is: LEADER
I20260619 08:22:34.662286 21207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:22:34.663101 21207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:22:34.663209 21207 catalog_manager.cc:1259] Loaded cluster ID: e57fa568d99342e880ca166e9c4ded89
I20260619 08:22:34.663246 21207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:22:34.664376 21207 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:22:34.664549 21207 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef: Loaded TSK: 0
I20260619 08:22:34.664599 21207 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ac68371830c44d3788301ebc2b4e80ef: Loaded TSK: 1
F20260619 08:22:34.664839 21207 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0de68ed9ad google::LogMessage::Fail() at ??:0
@ 0x7f0de68f088a google::LogMessage::SendToLog() at ??:0
@ 0x7f0de68ed4de google::LogMessage::Flush() at ??:0
@ 0x7f0de68eea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0de905a104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f0de9057d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f0dee8471cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f0dee84748e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f0dee847594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f0dee80dc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f0dee817ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0de87bb0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0de87b0954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0de96146db start_thread at ??:0
@ 0x7f0de53eb61f clone at ??:0
W20260619 08:22:34.768546 21199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:34.768853 21199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:34.768896 21199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:34.770730 21199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:34.770803 21199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:34.770848 21199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:22:34.773166 21199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:32805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=41251
--tserver_master_addrs=127.4.151.190:42845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:34.773500 21199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:34.773783 21199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:34.780375 21239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:34.780371 21238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:34.780889 21241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:34.780686 21199 server_base.cc:1034] running on GCE node
I20260619 08:22:34.781322 21199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:34.781786 21199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:34.783010 21199 hybrid_clock.cc:648] HybridClock initialized: now 1781857354782971 us; error 59 us; skew 500 ppm
Jun 19 08:22:34 dist-test-slave-d3n8 krb5kdc[20058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857354, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:22:34.785674 21199 init.cc:377] Logged in from keytab as kudu/127.4.151.129@KRBTEST.COM (short username kudu)
I20260619 08:22:34.786195 21199 webserver.cc:460] Webserver started at http://127.4.151.129:41251/ using document root <none> and password file <none>
I20260619 08:22:34.786446 21199 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:34.786518 21199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:34.788004 21199 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 08:22:34.788770 21248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:34.788946 21199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:22:34.789054 21199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "0d8f97fb52e048f79923c60a7e51c4cb"
format_stamp: "Formatted at 2026-06-19 08:21:52 on dist-test-slave-d3n8"
server_key: "03a8d733d06804c0d907afe8b26a8f1e"
server_key_iv: "eeda87f036f419bdeffc665a1c545bf1"
server_key_version: "encryptionkey@0"
I20260619 08:22:34.789467 21199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:34.825439 21199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:34.828015 21199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:34.828200 21199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:34.828459 21199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:22:34.828938 21255 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 08:22:34.831049 21199 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 08:22:34.831153 21199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 08:22:34.831313 21199 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 08:22:34.832021 21199 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 08:22:34.832084 21199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 08:22:34.832146 21255 tablet_bootstrap.cc:492] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb: Bootstrap starting.
I20260619 08:22:34.839594 21199 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:32805
I20260619 08:22:34.839758 21362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:32805 every 8 connection(s)
I20260619 08:22:34.840045 21199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
I20260619 08:22:34.843369 21255 log.cc:824] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:34.846278 21255 tablet_bootstrap.cc:492] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:22:34.846629 21255 tablet_bootstrap.cc:492] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb: Bootstrap complete.
I20260619 08:22:34.847018 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 21199
I20260619 08:22:34.847292 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:42405
--local_ip_for_outbound_sockets=127.4.151.130
--tserver_master_addrs=127.4.151.190:42845
--webserver_port=44915
--webserver_interface=127.4.151.130
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:22:34.847884 21255 ts_tablet_manager.cc:1397] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.010s
I20260619 08:22:34.849481 21255 raft_consensus.cc:348] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } }
I20260619 08:22:34.849830 21255 raft_consensus.cc:729] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d8f97fb52e048f79923c60a7e51c4cb, State: Initialized, Role: FOLLOWER
I20260619 08:22:34.849974 21255 consensus_queue.cc:260] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } }
I20260619 08:22:34.850276 21255 ts_tablet_manager.cc:1428] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb: Time spent starting tablet: real 0.002s user 0.003s sys 0.002s
W20260619 08:22:34.890439 21365 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:22:34.840823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.129:52373)
0619 08:22:34.841001 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:34.841046 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:34.841671 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:34.890219 (+ 48548us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":106,"threads_started":1}
W20260619 08:22:34.890725 21363 heartbeater.cc:644] Failed to heartbeat to 127.4.151.190:42845 (0 consecutive failures): Network error: Failed to ping master at 127.4.151.190:42845: Client connection negotiation failed: client connection to 127.4.151.190:42845: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 08:22:34.891093 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:34.890966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.129:60339)
0619 08:22:34.891024 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:34.891069 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 08:22:34.891489 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:34.891391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.129:53297)
0619 08:22:34.891437 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:34.891472 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 08:22:34.891605 21363 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 08:22:34.934427 21367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:34.934669 21367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:34.934711 21367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:34.935909 21367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:34.935976 21367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:34.936021 21367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:22:34.937606 21367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:42405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=44915
--tserver_master_addrs=127.4.151.190:42845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:34.937875 21367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:34.938107 21367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:34.943326 21374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:34.943442 21367 server_base.cc:1034] running on GCE node
W20260619 08:22:34.943326 21376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:34.943408 21373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:34.943827 21367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:34.944090 21367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:34.945262 21367 hybrid_clock.cc:648] HybridClock initialized: now 1781857354945233 us; error 43 us; skew 500 ppm
Jun 19 08:22:34 dist-test-slave-d3n8 krb5kdc[20058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857354, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:22:34.948158 21367 init.cc:377] Logged in from keytab as kudu/127.4.151.130@KRBTEST.COM (short username kudu)
I20260619 08:22:34.948598 21367 webserver.cc:460] Webserver started at http://127.4.151.130:44915/ using document root <none> and password file <none>
I20260619 08:22:34.948868 21367 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:34.948943 21367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:34.950373 21367 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:22:34.950976 21383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:34.951154 21367 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:22:34.951259 21367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a"
format_stamp: "Formatted at 2026-06-19 08:21:53 on dist-test-slave-d3n8"
server_key: "57bdb51ffc163a93e812d327133fa0cd"
server_key_iv: "0fbd96f440d5fa646955ee46d157b44f"
server_key_version: "encryptionkey@0"
I20260619 08:22:34.951608 21367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:34.983004 21367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:34.985337 21367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:34.985548 21367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:34.985817 21367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:22:34.986131 21367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:22:34.986186 21367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:34.986220 21367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:22:34.986249 21367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:34.992295 21367 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:42405
I20260619 08:22:34.992348 21496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:42405 every 8 connection(s)
I20260619 08:22:34.992681 21367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
I20260619 08:22:34.993587 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 21367
I20260619 08:22:34.993854 21499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:34.993360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.130:60649)
0619 08:22:34.993575 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:34.993656 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":98,"threads_started":1}
I20260619 08:22:34.993957 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:42699
--local_ip_for_outbound_sockets=127.4.151.131
--tserver_master_addrs=127.4.151.190:42845
--webserver_port=40871
--webserver_interface=127.4.151.131
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:22:34.994302 21497 heartbeater.cc:644] Failed to heartbeat to 127.4.151.190:42845 (0 consecutive failures): Network error: Failed to ping master at 127.4.151.190:42845: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
I20260619 08:22:34.994592 21499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:34.994496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.130:40333)
0619 08:22:34.994545 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:34.994570 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 08:22:34.994869 21499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:34.994778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.130:35077)
0619 08:22:34.994829 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:34.994850 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 08:22:34.995018 21497 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 08:22:35.079180 21500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:35.079551 21500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:35.079588 21500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:35.081306 21500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:35.081434 21500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:35.081485 21500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:22:35.083362 21500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:42699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=40871
--tserver_master_addrs=127.4.151.190:42845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:35.083585 21500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:35.083776 21500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:35.089216 21508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:35.089223 21506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:35.089300 21505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:35.089881 21500 server_base.cc:1034] running on GCE node
I20260619 08:22:35.090096 21500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:35.090379 21500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:35.091559 21500 hybrid_clock.cc:648] HybridClock initialized: now 1781857355091543 us; error 35 us; skew 500 ppm
Jun 19 08:22:35 dist-test-slave-d3n8 krb5kdc[20058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857355, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:22:35.094306 21500 init.cc:377] Logged in from keytab as kudu/127.4.151.131@KRBTEST.COM (short username kudu)
I20260619 08:22:35.094693 21500 webserver.cc:460] Webserver started at http://127.4.151.131:40871/ using document root <none> and password file <none>
I20260619 08:22:35.094882 21500 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:35.094924 21500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:35.096328 21500 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:22:35.096976 21515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:35.097254 21500 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:22:35.097340 21500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "1b64c563f60d4fb48895cbaaad57a82d"
format_stamp: "Formatted at 2026-06-19 08:21:53 on dist-test-slave-d3n8"
server_key: "f208fd339cf2bd3bb06eafea1ac11054"
server_key_iv: "eb4523c764e3871f9845a418035dba63"
server_key_version: "encryptionkey@0"
I20260619 08:22:35.097775 21500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:35.113554 21500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:35.116524 21500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:35.116739 21500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:35.117019 21500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:22:35.117574 21500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:22:35.117624 21500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:35.117652 21500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:22:35.117667 21500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:35.124140 21500 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:42699
I20260619 08:22:35.124272 21628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:42699 every 8 connection(s)
I20260619 08:22:35.124630 21500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
I20260619 08:22:35.125881 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:35.125468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.131:48977)
0619 08:22:35.125634 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:35.125711 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
W20260619 08:22:35.126432 21629 heartbeater.cc:644] Failed to heartbeat to 127.4.151.190:42845 (0 consecutive failures): Network error: Failed to ping master at 127.4.151.190:42845: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
I20260619 08:22:35.126796 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:35.126689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.131:43829)
0619 08:22:35.126743 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:35.126775 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 08:22:35.127235 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:35.127059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.131:33645)
0619 08:22:35.127133 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:35.127164 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 08:22:35.127444 21629 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 08:22:35.129850 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 21500
I20260619 08:22:35.892311 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:35.892064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.129:58477)
0619 08:22:35.892202 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:35.892246 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260619 08:22:35.995594 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:35.995395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.130:57991)
0619 08:22:35.995507 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:35.995543 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260619 08:22:36.128042 21635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:36.127792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.131:39395)
0619 08:22:36.127938 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:36.127980 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":59,"threads_started":1}
I20260619 08:22:36.469484 21636 raft_consensus.cc:482] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:22:36.469699 21636 raft_consensus.cc:504] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } }
I20260619 08:22:36.470166 21636 leader_election.cc:288] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b64c563f60d4fb48895cbaaad57a82d (127.4.151.131:42699), aa44be97d3d24fc2b1ff2ea93acf6f9a (127.4.151.130:42405)
Jun 19 08:22:36 dist-test-slave-d3n8 krb5kdc[20058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857354, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for kudu/127.4.151.131@KRBTEST.COM
Jun 19 08:22:36 dist-test-slave-d3n8 krb5kdc[20058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857354, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for kudu/127.4.151.130@KRBTEST.COM
I20260619 08:22:36.486227 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:36.470507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:52953 (local address 127.4.151.130:42405)
0619 08:22:36.470669 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:36.470677 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:36.470701 (+ 24us) server_negotiation.cc:407] Connection header received
0619 08:22:36.471131 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:36.471133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:36.471210 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:36.471357 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:36.472054 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:36.473161 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:36.474206 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:36.474652 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:36.474786 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:36.482729 (+ 7943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:36.482734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:36.482734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:22:36.482746 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:36.484527 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:36.484929 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:36.484931 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:36.484932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:36.485022 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:36.485241 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:36.485242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:36.485242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:36.485598 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:36.485699 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:36.485972 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 08:22:36.486103 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
I20260619 08:22:36.486210 21638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:36.470382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:42405 (local address 127.4.151.129:52953)
0619 08:22:36.470477 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:36.470489 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:36.470542 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:36.471348 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:36.471349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:36.471671 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:22:36.471876 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:36.471882 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:36.473270 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:36.473271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:36.474108 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:36.474110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:36.474759 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:36.474760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:36.474862 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:36.480290 (+ 5428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:22:36.480295 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:22:36.482527 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:22:36.484646 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:36.484647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:36.484648 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:36.484829 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:36.485100 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:36.485101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:36.485102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:36.485171 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:36.485679 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:22:36.485685 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:22:36.485813 (+ 128us) client_negotiation.cc:769] Sending connection context
0619 08:22:36.485923 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 08:22:36.486114 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":100,"thread_start_us":44,"threads_started":1}
I20260619 08:22:36.486253 21639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:36.470378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:47979 (local address 127.4.151.131:42699)
0619 08:22:36.470531 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:36.470542 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:36.470576 (+ 34us) server_negotiation.cc:407] Connection header received
0619 08:22:36.471035 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:36.471038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:36.471079 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:36.471193 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:36.472013 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:36.473027 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:36.474208 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:36.474640 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:36.474743 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:36.482223 (+ 7480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:36.482255 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:36.482255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:22:36.482269 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:36.484385 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:36.484853 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:36.484856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:36.484857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:36.484956 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:36.485242 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:36.485243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:36.485243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:36.485598 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:36.485701 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:36.485977 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 08:22:36.486135 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":52,"threads_started":1}
I20260619 08:22:36.486334 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:36.470260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:42699 (local address 127.4.151.129:47979)
0619 08:22:36.470395 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:36.470409 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:36.470474 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:36.471274 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:36.471277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:36.471659 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:22:36.471876 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:36.471882 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:36.473137 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:36.473138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:36.474109 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:36.474112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:36.474732 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:36.474733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:36.474865 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:36.480244 (+ 5379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:22:36.480262 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:22:36.482075 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:22:36.484506 (+ 2431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:36.484513 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:36.484516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:36.484759 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:36.485029 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:36.485030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:36.485030 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:36.485160 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:36.485683 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:22:36.485689 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:22:36.485812 (+ 123us) client_negotiation.cc:769] Sending connection context
0619 08:22:36.485924 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 08:22:36.486200 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":216,"thread_start_us":80,"threads_started":1}
W20260619 08:22:36.487648 21249 leader_election.cc:341] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b64c563f60d4fb48895cbaaad57a82d (127.4.151.131:42699): Not found: Tablet not found: ae1e202809864c0facdbf51e60e2e3b5
W20260619 08:22:36.487744 21252 leader_election.cc:341] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aa44be97d3d24fc2b1ff2ea93acf6f9a (127.4.151.130:42405): Not found: Tablet not found: ae1e202809864c0facdbf51e60e2e3b5
I20260619 08:22:36.487852 21252 leader_election.cc:302] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d8f97fb52e048f79923c60a7e51c4cb; no voters: 1b64c563f60d4fb48895cbaaad57a82d, aa44be97d3d24fc2b1ff2ea93acf6f9a
I20260619 08:22:36.487972 21636 raft_consensus.cc:2726] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 08:22:36.893041 21638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:36.892886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.129:58445)
0619 08:22:36.892956 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:36.893000 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 08:22:36.996337 21641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:36.996051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.130:51927)
0619 08:22:36.996257 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:36.996296 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":57,"threads_started":1}
I20260619 08:22:37.129014 21642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:37.128773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.131:37053)
0619 08:22:37.128923 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:37.128968 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":66,"threads_started":1}
I20260619 08:22:37.893872 21643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:37.893626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.129:53307)
0619 08:22:37.893776 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:37.893824 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260619 08:22:37.997228 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:37.996898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.130:56507)
0619 08:22:37.997051 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:37.997175 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260619 08:22:38.129885 21645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:38.129641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.131:33435)
0619 08:22:38.129787 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:38.129842 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
I20260619 08:22:38.319643 21646 raft_consensus.cc:482] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:22:38.319789 21646 raft_consensus.cc:504] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } }
I20260619 08:22:38.320005 21646 leader_election.cc:288] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b64c563f60d4fb48895cbaaad57a82d (127.4.151.131:42699), aa44be97d3d24fc2b1ff2ea93acf6f9a (127.4.151.130:42405)
W20260619 08:22:38.320503 21252 leader_election.cc:341] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aa44be97d3d24fc2b1ff2ea93acf6f9a (127.4.151.130:42405): Not found: Tablet not found: ae1e202809864c0facdbf51e60e2e3b5
W20260619 08:22:38.320629 21249 leader_election.cc:341] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b64c563f60d4fb48895cbaaad57a82d (127.4.151.131:42699): Not found: Tablet not found: ae1e202809864c0facdbf51e60e2e3b5
I20260619 08:22:38.320772 21249 leader_election.cc:302] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d8f97fb52e048f79923c60a7e51c4cb; no voters: 1b64c563f60d4fb48895cbaaad57a82d, aa44be97d3d24fc2b1ff2ea93acf6f9a
I20260619 08:22:38.320892 21646 raft_consensus.cc:2726] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 08:22:38.894663 21647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:38.894411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.129:35961)
0619 08:22:38.894563 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:38.894612 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":61,"threads_started":1}
I20260619 08:22:38.997982 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:38.997746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.130:52687)
0619 08:22:38.997895 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:38.997937 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":66,"threads_started":1}
I20260619 08:22:39.130769 21649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:39.130532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.131:41277)
0619 08:22:39.130665 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:39.130711 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260619 08:22:39.895520 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:39.895271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.129:46891)
0619 08:22:39.895421 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:39.895473 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":65,"threads_started":1}
I20260619 08:22:39.998776 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:39.998555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.130:40171)
0619 08:22:39.998686 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:39.998727 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20260619 08:22:40.131554 21652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:40.131334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.131:55629)
0619 08:22:40.131461 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:40.131506 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260619 08:22:40.577641 21653 raft_consensus.cc:482] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:22:40.577739 21653 raft_consensus.cc:504] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } }
I20260619 08:22:40.577973 21653 leader_election.cc:288] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b64c563f60d4fb48895cbaaad57a82d (127.4.151.131:42699), aa44be97d3d24fc2b1ff2ea93acf6f9a (127.4.151.130:42405)
W20260619 08:22:40.578379 21249 leader_election.cc:341] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b64c563f60d4fb48895cbaaad57a82d (127.4.151.131:42699): Not found: Tablet not found: ae1e202809864c0facdbf51e60e2e3b5
W20260619 08:22:40.578470 21252 leader_election.cc:341] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aa44be97d3d24fc2b1ff2ea93acf6f9a (127.4.151.130:42405): Not found: Tablet not found: ae1e202809864c0facdbf51e60e2e3b5
I20260619 08:22:40.578536 21252 leader_election.cc:302] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d8f97fb52e048f79923c60a7e51c4cb; no voters: 1b64c563f60d4fb48895cbaaad57a82d, aa44be97d3d24fc2b1ff2ea93acf6f9a
I20260619 08:22:40.578625 21653 raft_consensus.cc:2726] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 08:22:40.896412 21654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:40.896082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.129:39459)
0619 08:22:40.896313 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:40.896358 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":75,"threads_started":1}
I20260619 08:22:40.999557 21655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:40.999315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.130:54921)
0619 08:22:40.999463 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:40.999508 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":62,"threads_started":1}
I20260619 08:22:41.132469 21656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:41.132225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.131:53549)
0619 08:22:41.132368 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:41.132421 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260619 08:22:41.897318 21657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:41.897058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.129:36031)
0619 08:22:41.897206 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:41.897271 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":68,"threads_started":1}
I20260619 08:22:42.000517 21658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:42.000228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.130:53877)
0619 08:22:42.000424 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:42.000469 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":84,"threads_started":1}
I20260619 08:22:42.133196 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:42.132957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.131:47159)
0619 08:22:42.133107 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:42.133147 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":63,"threads_started":1}
I20260619 08:22:42.898180 21660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:42.897908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.129:49295)
0619 08:22:42.898087 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:42.898133 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":63,"threads_started":1}
I20260619 08:22:43.001454 21661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:43.001134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.130:43933)
0619 08:22:43.001311 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:43.001356 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":64,"threads_started":1}
I20260619 08:22:43.134011 21662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:43.133753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.131:58331)
0619 08:22:43.133925 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:43.133963 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":91,"threads_started":1}
I20260619 08:22:43.554363 21663 raft_consensus.cc:482] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:22:43.554477 21663 raft_consensus.cc:504] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b64c563f60d4fb48895cbaaad57a82d" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42699 } } peers { permanent_uuid: "aa44be97d3d24fc2b1ff2ea93acf6f9a" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 42405 } } peers { permanent_uuid: "0d8f97fb52e048f79923c60a7e51c4cb" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 32805 } }
I20260619 08:22:43.554723 21663 leader_election.cc:288] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b64c563f60d4fb48895cbaaad57a82d (127.4.151.131:42699), aa44be97d3d24fc2b1ff2ea93acf6f9a (127.4.151.130:42405)
W20260619 08:22:43.555186 21249 leader_election.cc:341] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b64c563f60d4fb48895cbaaad57a82d (127.4.151.131:42699): Not found: Tablet not found: ae1e202809864c0facdbf51e60e2e3b5
W20260619 08:22:43.555264 21252 leader_election.cc:341] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aa44be97d3d24fc2b1ff2ea93acf6f9a (127.4.151.130:42405): Not found: Tablet not found: ae1e202809864c0facdbf51e60e2e3b5
I20260619 08:22:43.555307 21252 leader_election.cc:302] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d8f97fb52e048f79923c60a7e51c4cb; no voters: 1b64c563f60d4fb48895cbaaad57a82d, aa44be97d3d24fc2b1ff2ea93acf6f9a
I20260619 08:22:43.555366 21663 raft_consensus.cc:2726] T ae1e202809864c0facdbf51e60e2e3b5 P 0d8f97fb52e048f79923c60a7e51c4cb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 08:22:43.899029 21664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:43.898774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.129:33341)
0619 08:22:43.898936 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:43.898982 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":73,"threads_started":1}
I20260619 08:22:44.002333 21665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:44.002035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.130:44189)
0619 08:22:44.002202 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:44.002254 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":64,"threads_started":1}
I20260619 08:22:44.134917 21666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:44.134597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.131:60719)
0619 08:22:44.134824 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:44.134869 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":113,"threads_started":1}
I20260619 08:22:44.899922 21667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:44.899677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.129:35761)
0619 08:22:44.899829 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:44.899876 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260619 08:22:45.003255 21668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:45.002952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.130:42601)
0619 08:22:45.003147 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:45.003206 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":82,"threads_started":1}
I20260619 08:22:45.135766 21669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:45.135516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.131:55099)
0619 08:22:45.135662 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:45.135706 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
I20260619 08:22:45.900897 21670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:45.900588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.129:40993)
0619 08:22:45.900805 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:45.900851 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":103,"threads_started":1}
I20260619 08:22:46.004164 21671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:46.003909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.130:43367)
0619 08:22:46.004062 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:46.004113 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":72,"threads_started":1}
I20260619 08:22:46.136670 21672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:46.136418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.131:36941)
0619 08:22:46.136569 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:46.136619 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260619 08:22:46.901938 21673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:46.901593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.129:36545)
0619 08:22:46.901779 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:46.901889 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":118,"threads_started":1}
I20260619 08:22:47.004982 21674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:47.004705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.130:58361)
0619 08:22:47.004882 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:47.004930 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":79,"threads_started":1}
I20260619 08:22:47.137651 21675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:47.137331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.131:38617)
0619 08:22:47.137509 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:47.137582 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":103,"threads_started":1}
I20260619 08:22:47.902737 21676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:47.902469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.129:42563)
0619 08:22:47.902625 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:47.902671 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":72,"threads_started":1}
I20260619 08:22:48.005824 21677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:48.005573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.130:52883)
0619 08:22:48.005724 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:48.005770 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":66,"threads_started":1}
I20260619 08:22:48.138419 21678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:48.138179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.131:45653)
0619 08:22:48.138317 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:48.138362 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
I20260619 08:22:48.903920 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:48.903529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.129:49615)
0619 08:22:48.903834 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:48.903873 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":229,"threads_started":1}
I20260619 08:22:49.006623 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:49.006413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.130:58483)
0619 08:22:49.006526 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:49.006570 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
I20260619 08:22:49.139158 21681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:49.138933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.131:60693)
0619 08:22:49.139062 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:49.139105 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260619 08:22:49.904844 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:49.904611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.129:37525)
0619 08:22:49.904755 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:49.904798 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":64,"threads_started":1}
I20260619 08:22:50.007613 21683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:50.007322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.130:52865)
0619 08:22:50.007517 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:50.007563 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":82,"threads_started":1}
I20260619 08:22:50.131117 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 21199
I20260619 08:22:50.137022 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 21367
I20260619 08:22:50.140028 21684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:50.139707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:42845 (local address 127.4.151.131:50499)
0619 08:22:50.139934 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:50.139984 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.190:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":71,"threads_started":1}
I20260619 08:22:50.142009 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 21500
2026-06-19T08:22:50Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58648 ms)
[----------] 30 tests from SecurityITest (239549 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:22:50 dist-test-slave-d3n8 krb5kdc[21687](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:22:50 dist-test-slave-d3n8 krb5kdc[21687](info): set up 2 sockets
Jun 19 08:22:50 dist-test-slave-d3n8 krb5kdc[21687](info): commencing operation
krb5kdc: starting...
W20260619 08:22:51.201124 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:22:51 dist-test-slave-d3n8 krb5kdc[21687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:22:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:22:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:22:51.330984 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:41569
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:44169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:41569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:22:51.428668 21700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:51.428900 21700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:51.428923 21700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:51.430190 21700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:22:51.430238 21700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:51.430251 21700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:51.430264 21700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:22:51.430276 21700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:22:51.431810 21700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:44169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:41569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:41569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:51.432085 21700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:51.432336 21700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:51.438436 21706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:51.438423 21700 server_base.cc:1034] running on GCE node
W20260619 08:22:51.438393 21705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:51.438393 21708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:51.438962 21700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:51.439339 21700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:51.440533 21700 hybrid_clock.cc:648] HybridClock initialized: now 1781857371440516 us; error 49 us; skew 500 ppm
Jun 19 08:22:51 dist-test-slave-d3n8 krb5kdc[21687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857371, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:22:51.443483 21700 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:22:51.443969 21700 webserver.cc:460] Webserver started at http://127.4.151.190:36063/ using document root <none> and password file <none>
I20260619 08:22:51.444254 21700 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:51.444335 21700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:51.444487 21700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:22:51.445675 21700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "89b0303685174b7b9dabef521fce1063"
format_stamp: "Formatted at 2026-06-19 08:22:51 on dist-test-slave-d3n8"
server_key: "3532b4707b05e7635bb8a81613c5d9e9"
server_key_iv: "c089b91eed3fbea24da222d5f7d070e8"
server_key_version: "encryptionkey@0"
I20260619 08:22:51.446064 21700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "89b0303685174b7b9dabef521fce1063"
format_stamp: "Formatted at 2026-06-19 08:22:51 on dist-test-slave-d3n8"
server_key: "3532b4707b05e7635bb8a81613c5d9e9"
server_key_iv: "c089b91eed3fbea24da222d5f7d070e8"
server_key_version: "encryptionkey@0"
I20260619 08:22:51.447518 21700 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:22:51.448328 21715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:51.448577 21700 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:22:51.448697 21700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "89b0303685174b7b9dabef521fce1063"
format_stamp: "Formatted at 2026-06-19 08:22:51 on dist-test-slave-d3n8"
server_key: "3532b4707b05e7635bb8a81613c5d9e9"
server_key_iv: "c089b91eed3fbea24da222d5f7d070e8"
server_key_version: "encryptionkey@0"
I20260619 08:22:51.448766 21700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:51.459421 21700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:51.461890 21700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:51.462065 21700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:51.466640 21700 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:41569
I20260619 08:22:51.466684 21767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:41569 every 8 connection(s)
I20260619 08:22:51.467043 21700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:22:51.467693 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 21700
I20260619 08:22:51.467813 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:22:51.468020 4702 external_mini_cluster.cc:1428] Setting key 1f189e5a512fcd497192823c39eff3c3
I20260619 08:22:51.468052 21768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:51.471210 21768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89b0303685174b7b9dabef521fce1063: Bootstrap starting.
I20260619 08:22:51.471817 21768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89b0303685174b7b9dabef521fce1063: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:51.472162 21768 log.cc:824] T 00000000000000000000000000000000 P 89b0303685174b7b9dabef521fce1063: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:51.472937 21768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89b0303685174b7b9dabef521fce1063: No bootstrap required, opened a new log
Jun 19 08:22:51 dist-test-slave-d3n8 krb5kdc[21687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:22:51.475538 21768 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89b0303685174b7b9dabef521fce1063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b0303685174b7b9dabef521fce1063" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 41569 } }
I20260619 08:22:51.475733 21768 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89b0303685174b7b9dabef521fce1063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:51.475783 21768 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89b0303685174b7b9dabef521fce1063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89b0303685174b7b9dabef521fce1063, State: Initialized, Role: FOLLOWER
I20260619 08:22:51.475934 21768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89b0303685174b7b9dabef521fce1063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b0303685174b7b9dabef521fce1063" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 41569 } }
I20260619 08:22:51.476029 21768 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89b0303685174b7b9dabef521fce1063 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:22:51.476068 21768 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89b0303685174b7b9dabef521fce1063 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:22:51.476115 21768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89b0303685174b7b9dabef521fce1063 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:51.476831 21768 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89b0303685174b7b9dabef521fce1063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b0303685174b7b9dabef521fce1063" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 41569 } }
I20260619 08:22:51.476925 21768 leader_election.cc:302] T 00000000000000000000000000000000 P 89b0303685174b7b9dabef521fce1063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89b0303685174b7b9dabef521fce1063; no voters:
I20260619 08:22:51.477147 21768 leader_election.cc:288] T 00000000000000000000000000000000 P 89b0303685174b7b9dabef521fce1063 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:22:51.477178 21773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89b0303685174b7b9dabef521fce1063 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:22:51.477303 21773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89b0303685174b7b9dabef521fce1063 [term 1 LEADER]: Becoming Leader. State: Replica: 89b0303685174b7b9dabef521fce1063, State: Running, Role: LEADER
I20260619 08:22:51.477412 21773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89b0303685174b7b9dabef521fce1063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b0303685174b7b9dabef521fce1063" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 41569 } }
I20260619 08:22:51.477469 21768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89b0303685174b7b9dabef521fce1063 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:22:51.477734 21773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89b0303685174b7b9dabef521fce1063 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89b0303685174b7b9dabef521fce1063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b0303685174b7b9dabef521fce1063" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 41569 } } }
I20260619 08:22:51.477998 21773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89b0303685174b7b9dabef521fce1063 [sys.catalog]: This master's current role is: LEADER
I20260619 08:22:51.478236 21773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89b0303685174b7b9dabef521fce1063 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89b0303685174b7b9dabef521fce1063. Latest consensus state: current_term: 1 leader_uuid: "89b0303685174b7b9dabef521fce1063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b0303685174b7b9dabef521fce1063" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 41569 } } }
I20260619 08:22:51.478315 21782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:22:51.478348 21773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89b0303685174b7b9dabef521fce1063 [sys.catalog]: This master's current role is: LEADER
I20260619 08:22:51.478920 21782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:22:51.480011 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:51.468584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36804 (local address 127.4.151.190:41569)
0619 08:22:51.468787 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:51.468794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:51.468819 (+ 25us) server_negotiation.cc:407] Connection header received
0619 08:22:51.469220 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:51.469226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:51.469271 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:51.469447 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:51.470081 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:51.470976 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:51.471846 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:51.472355 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:51.472475 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:51.475514 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:51.475521 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:51.475522 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:22:51.475538 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:51.477911 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:51.478361 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:51.478364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:51.478365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:51.478469 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:51.478700 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:51.478701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:51.478701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:51.479025 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:51.479172 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:51.479762 (+ 590us) server_negotiation.cc:299] Negotiation successful
0619 08:22:51.479868 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20260619 08:22:51.480567 21782 catalog_manager.cc:1347] Generated new cluster ID: c594586b65cc4c7a876d966fa06163a6
I20260619 08:22:51.480616 21782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:22:51.485728 21782 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:22:51.486315 21782 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:22:51.497731 21782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89b0303685174b7b9dabef521fce1063: Generated new TSK 0
I20260619 08:22:51.497967 21782 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.151.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
I20260619 08:22:51.533041 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:41569
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:44169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:22:51.618933 21794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:51.619140 21794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:51.619164 21794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:51.620373 21794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:51.620416 21794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:51.620445 21794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:22:51.621949 21794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:44169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:41569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:51.622195 21794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:51.622448 21794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:51.627671 21802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:51.627678 21800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:51.627671 21799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:51.627866 21794 server_base.cc:1034] running on GCE node
I20260619 08:22:51.628414 21794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:51.628677 21794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:51.629869 21794 hybrid_clock.cc:648] HybridClock initialized: now 1781857371629842 us; error 46 us; skew 500 ppm
Jun 19 08:22:51 dist-test-slave-d3n8 krb5kdc[21687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857371, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:22:51.632371 21794 init.cc:377] Logged in from keytab as kudu/127.4.151.129@KRBTEST.COM (short username kudu)
I20260619 08:22:51.632891 21794 webserver.cc:460] Webserver started at http://127.4.151.129:37803/ using document root <none> and password file <none>
I20260619 08:22:51.633082 21794 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:51.633140 21794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:51.633231 21794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:22:51.634421 21794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "f6a13f10e05b49579e21acf15ddab6bc"
format_stamp: "Formatted at 2026-06-19 08:22:51 on dist-test-slave-d3n8"
server_key: "7dc59fbfb3e06d583144e0a433cbb0ba"
server_key_iv: "10b476a617d7a1004cf6578d9604544f"
server_key_version: "encryptionkey@0"
I20260619 08:22:51.634723 21794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "f6a13f10e05b49579e21acf15ddab6bc"
format_stamp: "Formatted at 2026-06-19 08:22:51 on dist-test-slave-d3n8"
server_key: "7dc59fbfb3e06d583144e0a433cbb0ba"
server_key_iv: "10b476a617d7a1004cf6578d9604544f"
server_key_version: "encryptionkey@0"
I20260619 08:22:51.636000 21794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:22:51.636725 21809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:51.636965 21794 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:22:51.637036 21794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "f6a13f10e05b49579e21acf15ddab6bc"
format_stamp: "Formatted at 2026-06-19 08:22:51 on dist-test-slave-d3n8"
server_key: "7dc59fbfb3e06d583144e0a433cbb0ba"
server_key_iv: "10b476a617d7a1004cf6578d9604544f"
server_key_version: "encryptionkey@0"
I20260619 08:22:51.637091 21794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:51.669458 21794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:51.671988 21794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:51.672106 21794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:51.672290 21794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:22:51.672642 21794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:22:51.672679 21794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:51.672704 21794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:22:51.672760 21794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:51.678916 21794 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:39335
I20260619 08:22:51.678964 21922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:39335 every 8 connection(s)
I20260619 08:22:51.679342 21794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
Jun 19 08:22:51 dist-test-slave-d3n8 krb5kdc[21687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857371, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:22:51.688591 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 21794
I20260619 08:22:51.688711 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:22:51.688947 4702 external_mini_cluster.cc:1428] Setting key 57efb59599ca47721b6eca8e19e19a90
I20260619 08:22:51.688953 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:51.680081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:57197 (local address 127.4.151.190:41569)
0619 08:22:51.680153 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:51.680156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:51.680375 (+ 219us) server_negotiation.cc:407] Connection header received
0619 08:22:51.680851 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:51.680852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:51.680866 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:51.680932 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:51.681906 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:51.682429 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:51.683486 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:51.683717 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:51.683786 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:51.686264 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:51.686265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:51.686265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:22:51.686272 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:51.687531 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:51.687934 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:51.687935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:51.687936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:51.687998 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:51.688237 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:51.688238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:51.688238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:51.688356 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:51.688417 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:51.688789 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 08:22:51.688885 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 08:22:51.689188 21925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:51.680069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:41569 (local address 127.4.151.129:57197)
0619 08:22:51.680237 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:51.680281 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:51.680767 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:51.680980 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:51.680984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:51.681578 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:22:51.681784 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:51.681789 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:51.682522 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:51.682522 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:51.683382 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:51.683384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:51.683809 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:51.683810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:51.683918 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:51.684679 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:22:51.684687 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:22:51.686157 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:22:51.687610 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:51.687617 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:51.687619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:51.687846 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:51.688081 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:51.688082 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:51.688082 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:51.688179 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:51.688430 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:22:51.688437 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:22:51.688609 (+ 172us) client_negotiation.cc:769] Sending connection context
0619 08:22:51.688738 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 08:22:51.688958 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260619 08:22:51.689810 21923 heartbeater.cc:347] Connected to a master server at 127.4.151.190:41569
I20260619 08:22:51.689927 21923 heartbeater.cc:464] Registering TS with master...
I20260619 08:22:51.690133 21923 heartbeater.cc:511] Master 127.4.151.190:41569 requested a full tablet report, sending...
I20260619 08:22:51.690670 21732 ts_manager.cc:194] Registered new tserver with Master: f6a13f10e05b49579e21acf15ddab6bc (127.4.151.129:39335)
I20260619 08:22:51.691522 21732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.129@KRBTEST.COM'} at 127.4.151.129:57197
WARNING: no policy specified for kudu/127.4.151.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
I20260619 08:22:51.716622 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:41569
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:44169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:22:51.798067 21928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:51.798345 21928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:51.798385 21928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:51.799604 21928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:51.799698 21928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:51.799747 21928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:22:51.801306 21928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:44169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:41569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:51.801620 21928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:51.801872 21928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:51.807370 21934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:51.807387 21936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:51.807379 21933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:51.807420 21928 server_base.cc:1034] running on GCE node
I20260619 08:22:51.807868 21928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:51.808084 21928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:51.809340 21928 hybrid_clock.cc:648] HybridClock initialized: now 1781857371809314 us; error 46 us; skew 500 ppm
Jun 19 08:22:51 dist-test-slave-d3n8 krb5kdc[21687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857371, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:22:51.812307 21928 init.cc:377] Logged in from keytab as kudu/127.4.151.130@KRBTEST.COM (short username kudu)
I20260619 08:22:51.812789 21928 webserver.cc:460] Webserver started at http://127.4.151.130:33545/ using document root <none> and password file <none>
I20260619 08:22:51.813042 21928 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:51.813119 21928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:51.813280 21928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:22:51.814601 21928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "ca97332580e949128666405b87d6ff4f"
format_stamp: "Formatted at 2026-06-19 08:22:51 on dist-test-slave-d3n8"
server_key: "ad0384d9aff306e0860b04429846633f"
server_key_iv: "bf22193aba7ba70f26ecd6889c080318"
server_key_version: "encryptionkey@0"
I20260619 08:22:51.814971 21928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "ca97332580e949128666405b87d6ff4f"
format_stamp: "Formatted at 2026-06-19 08:22:51 on dist-test-slave-d3n8"
server_key: "ad0384d9aff306e0860b04429846633f"
server_key_iv: "bf22193aba7ba70f26ecd6889c080318"
server_key_version: "encryptionkey@0"
I20260619 08:22:51.816391 21928 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:22:51.817211 21943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:51.817456 21928 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:22:51.817560 21928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "ca97332580e949128666405b87d6ff4f"
format_stamp: "Formatted at 2026-06-19 08:22:51 on dist-test-slave-d3n8"
server_key: "ad0384d9aff306e0860b04429846633f"
server_key_iv: "bf22193aba7ba70f26ecd6889c080318"
server_key_version: "encryptionkey@0"
I20260619 08:22:51.817660 21928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:51.837114 21928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:51.839974 21928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:51.840139 21928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:51.840404 21928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:22:51.840771 21928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:22:51.840824 21928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:51.840860 21928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:22:51.840889 21928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:51.846802 21928 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:37191
I20260619 08:22:51.846879 22056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:37191 every 8 connection(s)
I20260619 08:22:51.847204 21928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
I20260619 08:22:51.851930 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 21928
I20260619 08:22:51.852074 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:22:51.852298 4702 external_mini_cluster.cc:1428] Setting key 8729aef385d92ccaac212e68b26c4915
Jun 19 08:22:51 dist-test-slave-d3n8 krb5kdc[21687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857371, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:22:51.857954 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:51.848037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:45345 (local address 127.4.151.190:41569)
0619 08:22:51.848101 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:51.848103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:51.848303 (+ 200us) server_negotiation.cc:407] Connection header received
0619 08:22:51.848854 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:51.848855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:51.848868 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:51.848913 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:51.849928 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:51.850446 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:51.851562 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:51.851808 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:51.851885 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:51.854587 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:51.854588 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:51.854589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:22:51.854596 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:51.856321 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:51.856882 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:51.856883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:51.856884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:51.856950 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:51.857235 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:51.857236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:51.857236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:51.857363 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:51.857474 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:51.857766 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 08:22:51.857861 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 08:22:51.858151 22059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:51.847993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:41569 (local address 127.4.151.130:45345)
0619 08:22:51.848153 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:51.848201 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:51.848743 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:51.848975 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:51.848980 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:51.849625 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:22:51.849827 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:51.849831 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:51.850545 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:51.850546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:51.851459 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:51.851461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:51.851921 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:51.851922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:51.852126 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:51.852851 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:22:51.852859 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:22:51.854491 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:22:51.856475 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:51.856482 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:51.856483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:51.856749 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:51.857031 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:51.857031 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:51.857032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:51.857156 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:51.857464 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:22:51.857471 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:22:51.857622 (+ 151us) client_negotiation.cc:769] Sending connection context
0619 08:22:51.857733 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 08:22:51.857907 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260619 08:22:51.858712 22057 heartbeater.cc:347] Connected to a master server at 127.4.151.190:41569
I20260619 08:22:51.858827 22057 heartbeater.cc:464] Registering TS with master...
I20260619 08:22:51.859025 22057 heartbeater.cc:511] Master 127.4.151.190:41569 requested a full tablet report, sending...
I20260619 08:22:51.859441 21732 ts_manager.cc:194] Registered new tserver with Master: ca97332580e949128666405b87d6ff4f (127.4.151.130:37191)
I20260619 08:22:51.859895 21732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.130@KRBTEST.COM'} at 127.4.151.130:45345
WARNING: no policy specified for kudu/127.4.151.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
I20260619 08:22:51.881613 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:41569
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:44169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:22:51.967788 22062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:51.968015 22062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:51.968041 22062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:51.969166 22062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:51.969204 22062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:51.969230 22062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:22:51.970670 22062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:44169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:41569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:51.970877 22062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:51.971066 22062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:51.976781 22068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:51.976840 22067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:51.976925 22062 server_base.cc:1034] running on GCE node
W20260619 08:22:51.976791 22070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:51.977423 22062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:51.977694 22062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:51.978859 22062 hybrid_clock.cc:648] HybridClock initialized: now 1781857371978845 us; error 39 us; skew 500 ppm
Jun 19 08:22:51 dist-test-slave-d3n8 krb5kdc[21687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857371, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:22:51.981853 22062 init.cc:377] Logged in from keytab as kudu/127.4.151.131@KRBTEST.COM (short username kudu)
I20260619 08:22:51.982331 22062 webserver.cc:460] Webserver started at http://127.4.151.131:46067/ using document root <none> and password file <none>
I20260619 08:22:51.982579 22062 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:51.982651 22062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:51.982800 22062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:22:51.983925 22062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "15482df87c4a4487bbb49c9728dbbd4c"
format_stamp: "Formatted at 2026-06-19 08:22:51 on dist-test-slave-d3n8"
server_key: "fc506d3fd63c29b29a4008b6c9357e25"
server_key_iv: "fd89004630b368845bb576de10b622aa"
server_key_version: "encryptionkey@0"
I20260619 08:22:51.984334 22062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "15482df87c4a4487bbb49c9728dbbd4c"
format_stamp: "Formatted at 2026-06-19 08:22:51 on dist-test-slave-d3n8"
server_key: "fc506d3fd63c29b29a4008b6c9357e25"
server_key_iv: "fd89004630b368845bb576de10b622aa"
server_key_version: "encryptionkey@0"
I20260619 08:22:51.985832 22062 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:22:51.986718 22077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:51.987011 22062 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:22:51.987123 22062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "15482df87c4a4487bbb49c9728dbbd4c"
format_stamp: "Formatted at 2026-06-19 08:22:51 on dist-test-slave-d3n8"
server_key: "fc506d3fd63c29b29a4008b6c9357e25"
server_key_iv: "fd89004630b368845bb576de10b622aa"
server_key_version: "encryptionkey@0"
I20260619 08:22:51.987206 22062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:52.002324 22062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:52.005084 22062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:52.005264 22062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:52.005556 22062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:22:52.005885 22062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:22:52.005942 22062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:52.005988 22062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:22:52.006012 22062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:52.012077 22062 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:33729
I20260619 08:22:52.012158 22190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:33729 every 8 connection(s)
I20260619 08:22:52.012496 22062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
I20260619 08:22:52.017617 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 22062
I20260619 08:22:52.017724 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:22:52.017937 4702 external_mini_cluster.cc:1428] Setting key d67a4715fc160398b06a229ce31f540f
Jun 19 08:22:52 dist-test-slave-d3n8 krb5kdc[21687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857371, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:22:52.022957 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.013311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:48173 (local address 127.4.151.190:41569)
0619 08:22:52.013425 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:52.013428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:52.013616 (+ 188us) server_negotiation.cc:407] Connection header received
0619 08:22:52.014163 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:52.014164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:52.014177 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:52.014223 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:52.015381 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.015897 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.016914 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.017140 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.017216 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.019786 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:52.019787 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:52.019788 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:22:52.019796 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:52.021525 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:52.021959 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:52.021960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:52.021961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:52.022025 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:52.022290 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:52.022290 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:52.022291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:52.022415 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:52.022493 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:52.022779 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 08:22:52.022864 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 08:22:52.023157 22193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.013268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:41569 (local address 127.4.151.131:48173)
0619 08:22:52.013470 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:52.013519 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:52.014037 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:52.014288 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:52.014292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:52.015070 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:22:52.015268 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:52.015273 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.015996 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.015997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:52.016817 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:52.016819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.017237 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.017238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:52.017377 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.018005 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:22:52.018016 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:22:52.019665 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:22:52.021611 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:52.021618 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:52.021620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:52.021874 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:52.022097 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:52.022098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:52.022098 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:52.022219 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:52.022483 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:22:52.022490 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:22:52.022636 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 08:22:52.022748 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 08:22:52.022918 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":108,"threads_started":1}
I20260619 08:22:52.023723 22191 heartbeater.cc:347] Connected to a master server at 127.4.151.190:41569
I20260619 08:22:52.023808 22191 heartbeater.cc:464] Registering TS with master...
I20260619 08:22:52.024015 22191 heartbeater.cc:511] Master 127.4.151.190:41569 requested a full tablet report, sending...
I20260619 08:22:52.024454 21732 ts_manager.cc:194] Registered new tserver with Master: 15482df87c4a4487bbb49c9728dbbd4c (127.4.151.131:33729)
I20260619 08:22:52.024880 21732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.131@KRBTEST.COM'} at 127.4.151.131:48173
I20260619 08:22:52.030059 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:22:52.038018 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.031496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36816 (local address 127.4.151.190:41569)
0619 08:22:52.031568 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:52.031570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:52.031685 (+ 115us) server_negotiation.cc:407] Connection header received
0619 08:22:52.031765 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:52.031766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:52.031776 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:52.031827 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:52.032403 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.032910 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.033741 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.033943 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.034027 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.035317 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:52.035343 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:52.035343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:22:52.035350 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:52.036686 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:52.037065 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:52.037066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:52.037067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:52.037135 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:52.037394 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:52.037395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:52.037396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:52.037562 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:52.037690 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:52.037823 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 08:22:52.037912 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 08:22:52.040047 21732 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:22:52.042272 21732 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:22:52.049845 22207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.045739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56022 (local address 127.4.151.129:39335)
0619 08:22:52.045914 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:52.045919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:52.045929 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:22:52.045972 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:52.045974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:52.045989 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:52.046254 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:52.046330 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.047036 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.048667 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.049362 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.049458 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.049525 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:52.049712 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 08:22:52.049756 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20260619 08:22:52.049885 22204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.045182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:39335 (local address 127.0.0.1:56022)
0619 08:22:52.045398 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:52.045445 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:52.045502 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:52.046091 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:52.046091 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:52.046093 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:52.046179 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:52.046181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.047152 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.047153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:52.048588 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:52.048591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.049482 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.049483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:52.049586 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.049595 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 08:22:52.049683 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 08:22:52.049778 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":60,"thread_start_us":125,"threads_started":1}
I20260619 08:22:52.049912 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.045497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34972 (local address 127.4.151.131:33729)
0619 08:22:52.045655 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:52.045660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:52.045995 (+ 335us) server_negotiation.cc:407] Connection header received
0619 08:22:52.046117 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:52.046119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:52.046133 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:52.046187 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:52.046411 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.047449 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.048576 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.049340 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.049469 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.049525 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:52.049712 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 08:22:52.049756 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
I20260619 08:22:52.050987 21857 tablet_service.cc:1467] Processing CreateTablet for tablet 4a7cb438427c440d996e9e50f2ee9834 (DEFAULT_TABLE table=test-table [id=1a3a0c21440a4c358a83c58c4e70df46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:22:52.051229 22205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.045392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35906 (local address 127.4.151.130:37191)
0619 08:22:52.045604 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:52.045609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:52.045619 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:22:52.045667 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:52.045668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:52.045680 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:52.045790 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:52.045970 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.047122 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.049732 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.050800 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.050881 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.050937 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:52.051093 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 08:22:52.051139 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":97,"threads_started":1}
I20260619 08:22:52.051204 22202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.045144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:37191 (local address 127.0.0.1:35906)
0619 08:22:52.045347 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:52.045416 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:52.045492 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:52.045785 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:52.045788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:52.045790 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:52.045884 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:52.045886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.047217 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.047218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:52.048589 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:52.048591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.050905 (+ 2314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.050906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:52.050976 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.050980 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:22:52.051072 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 08:22:52.051124 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":121,"thread_start_us":66,"threads_started":1}
I20260619 08:22:52.051453 21857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a7cb438427c440d996e9e50f2ee9834. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:52.051926 21991 tablet_service.cc:1467] Processing CreateTablet for tablet 4a7cb438427c440d996e9e50f2ee9834 (DEFAULT_TABLE table=test-table [id=1a3a0c21440a4c358a83c58c4e70df46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:22:52.052219 21991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a7cb438427c440d996e9e50f2ee9834. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:52.054380 22208 tablet_bootstrap.cc:492] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc: Bootstrap starting.
I20260619 08:22:52.049885 22203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.045230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:33729 (local address 127.0.0.1:34972)
0619 08:22:52.045924 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:52.045935 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:52.046024 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:52.046281 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:52.046282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:52.046283 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:52.046327 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:52.046328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.047559 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.047560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:52.048481 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:52.048484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.049481 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.049482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:52.049579 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.049595 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 08:22:52.049683 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 08:22:52.049777 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":2,"thread_start_us":90,"threads_started":1}
I20260619 08:22:52.055115 22208 tablet_bootstrap.cc:654] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:52.055308 22209 tablet_bootstrap.cc:492] T 4a7cb438427c440d996e9e50f2ee9834 P ca97332580e949128666405b87d6ff4f: Bootstrap starting.
I20260619 08:22:52.055202 22125 tablet_service.cc:1467] Processing CreateTablet for tablet 4a7cb438427c440d996e9e50f2ee9834 (DEFAULT_TABLE table=test-table [id=1a3a0c21440a4c358a83c58c4e70df46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:22:52.055570 22125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a7cb438427c440d996e9e50f2ee9834. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:52.055688 22208 log.cc:824] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:52.056033 22209 tablet_bootstrap.cc:654] T 4a7cb438427c440d996e9e50f2ee9834 P ca97332580e949128666405b87d6ff4f: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:52.056571 22208 tablet_bootstrap.cc:492] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc: No bootstrap required, opened a new log
I20260619 08:22:52.056571 22209 log.cc:824] T 4a7cb438427c440d996e9e50f2ee9834 P ca97332580e949128666405b87d6ff4f: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:52.056649 22208 ts_tablet_manager.cc:1397] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:22:52.057468 22209 tablet_bootstrap.cc:492] T 4a7cb438427c440d996e9e50f2ee9834 P ca97332580e949128666405b87d6ff4f: No bootstrap required, opened a new log
I20260619 08:22:52.057545 22209 ts_tablet_manager.cc:1397] T 4a7cb438427c440d996e9e50f2ee9834 P ca97332580e949128666405b87d6ff4f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:22:52.058869 22212 tablet_bootstrap.cc:492] T 4a7cb438427c440d996e9e50f2ee9834 P 15482df87c4a4487bbb49c9728dbbd4c: Bootstrap starting.
I20260619 08:22:52.058808 22208 raft_consensus.cc:348] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97332580e949128666405b87d6ff4f" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37191 } } peers { permanent_uuid: "f6a13f10e05b49579e21acf15ddab6bc" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 39335 } } peers { permanent_uuid: "15482df87c4a4487bbb49c9728dbbd4c" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33729 } }
I20260619 08:22:52.059005 22208 raft_consensus.cc:374] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:52.059060 22208 raft_consensus.cc:729] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a13f10e05b49579e21acf15ddab6bc, State: Initialized, Role: FOLLOWER
I20260619 08:22:52.058990 22209 raft_consensus.cc:348] T 4a7cb438427c440d996e9e50f2ee9834 P ca97332580e949128666405b87d6ff4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97332580e949128666405b87d6ff4f" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37191 } } peers { permanent_uuid: "f6a13f10e05b49579e21acf15ddab6bc" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 39335 } } peers { permanent_uuid: "15482df87c4a4487bbb49c9728dbbd4c" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33729 } }
I20260619 08:22:52.059170 22209 raft_consensus.cc:374] T 4a7cb438427c440d996e9e50f2ee9834 P ca97332580e949128666405b87d6ff4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:52.059185 22208 consensus_queue.cc:260] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97332580e949128666405b87d6ff4f" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37191 } } peers { permanent_uuid: "f6a13f10e05b49579e21acf15ddab6bc" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 39335 } } peers { permanent_uuid: "15482df87c4a4487bbb49c9728dbbd4c" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33729 } }
I20260619 08:22:52.059222 22209 raft_consensus.cc:729] T 4a7cb438427c440d996e9e50f2ee9834 P ca97332580e949128666405b87d6ff4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca97332580e949128666405b87d6ff4f, State: Initialized, Role: FOLLOWER
I20260619 08:22:52.059329 22209 consensus_queue.cc:260] T 4a7cb438427c440d996e9e50f2ee9834 P ca97332580e949128666405b87d6ff4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97332580e949128666405b87d6ff4f" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37191 } } peers { permanent_uuid: "f6a13f10e05b49579e21acf15ddab6bc" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 39335 } } peers { permanent_uuid: "15482df87c4a4487bbb49c9728dbbd4c" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33729 } }
I20260619 08:22:52.059468 22208 ts_tablet_manager.cc:1428] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:22:52.059479 21923 heartbeater.cc:503] Master 127.4.151.190:41569 was elected leader, sending a full tablet report...
I20260619 08:22:52.059612 22209 ts_tablet_manager.cc:1428] T 4a7cb438427c440d996e9e50f2ee9834 P ca97332580e949128666405b87d6ff4f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:22:52.059777 22057 heartbeater.cc:503] Master 127.4.151.190:41569 was elected leader, sending a full tablet report...
I20260619 08:22:52.059998 22212 tablet_bootstrap.cc:654] T 4a7cb438427c440d996e9e50f2ee9834 P 15482df87c4a4487bbb49c9728dbbd4c: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:52.060551 22212 log.cc:824] T 4a7cb438427c440d996e9e50f2ee9834 P 15482df87c4a4487bbb49c9728dbbd4c: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:52.061400 22212 tablet_bootstrap.cc:492] T 4a7cb438427c440d996e9e50f2ee9834 P 15482df87c4a4487bbb49c9728dbbd4c: No bootstrap required, opened a new log
I20260619 08:22:52.061468 22212 ts_tablet_manager.cc:1397] T 4a7cb438427c440d996e9e50f2ee9834 P 15482df87c4a4487bbb49c9728dbbd4c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 08:22:52.062857 22212 raft_consensus.cc:348] T 4a7cb438427c440d996e9e50f2ee9834 P 15482df87c4a4487bbb49c9728dbbd4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97332580e949128666405b87d6ff4f" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37191 } } peers { permanent_uuid: "f6a13f10e05b49579e21acf15ddab6bc" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 39335 } } peers { permanent_uuid: "15482df87c4a4487bbb49c9728dbbd4c" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33729 } }
I20260619 08:22:52.063019 22212 raft_consensus.cc:374] T 4a7cb438427c440d996e9e50f2ee9834 P 15482df87c4a4487bbb49c9728dbbd4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:52.063066 22212 raft_consensus.cc:729] T 4a7cb438427c440d996e9e50f2ee9834 P 15482df87c4a4487bbb49c9728dbbd4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15482df87c4a4487bbb49c9728dbbd4c, State: Initialized, Role: FOLLOWER
I20260619 08:22:52.063217 22212 consensus_queue.cc:260] T 4a7cb438427c440d996e9e50f2ee9834 P 15482df87c4a4487bbb49c9728dbbd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97332580e949128666405b87d6ff4f" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37191 } } peers { permanent_uuid: "f6a13f10e05b49579e21acf15ddab6bc" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 39335 } } peers { permanent_uuid: "15482df87c4a4487bbb49c9728dbbd4c" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33729 } }
I20260619 08:22:52.063488 22212 ts_tablet_manager.cc:1428] T 4a7cb438427c440d996e9e50f2ee9834 P 15482df87c4a4487bbb49c9728dbbd4c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:22:52.063529 22191 heartbeater.cc:503] Master 127.4.151.190:41569 was elected leader, sending a full tablet report...
W20260619 08:22:52.097870 22058 tablet.cc:2369] T 4a7cb438427c440d996e9e50f2ee9834 P ca97332580e949128666405b87d6ff4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:22:52.180096 21924 tablet.cc:2369] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:22:52.223834 22213 raft_consensus.cc:482] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:22:52.224016 22213 raft_consensus.cc:504] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97332580e949128666405b87d6ff4f" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37191 } } peers { permanent_uuid: "f6a13f10e05b49579e21acf15ddab6bc" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 39335 } } peers { permanent_uuid: "15482df87c4a4487bbb49c9728dbbd4c" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33729 } }
I20260619 08:22:52.224462 22213 leader_election.cc:288] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca97332580e949128666405b87d6ff4f (127.4.151.130:37191), 15482df87c4a4487bbb49c9728dbbd4c (127.4.151.131:33729)
I20260619 08:22:52.227705 22205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.224739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:44793 (local address 127.4.151.130:37191)
0619 08:22:52.224805 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:52.224807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:52.224859 (+ 52us) server_negotiation.cc:407] Connection header received
0619 08:22:52.224953 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:52.224954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:52.224971 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:52.225021 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:52.225216 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.225769 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.226795 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.227330 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.227390 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.227398 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:52.227580 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 08:22:52.227626 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 08:22:52.227730 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.224790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:59229 (local address 127.4.151.131:33729)
0619 08:22:52.224837 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:52.224839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:52.224884 (+ 45us) server_negotiation.cc:407] Connection header received
0619 08:22:52.224973 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:52.224974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:52.224989 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:52.225062 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:52.225264 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.225788 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.226856 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.227342 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.227437 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.227446 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:52.227623 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 08:22:52.227663 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 08:22:52.227746 22217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.224653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:37191 (local address 127.4.151.129:44793)
0619 08:22:52.224794 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:52.224806 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:52.224877 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:52.225044 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:52.225045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:52.225048 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:52.225138 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:52.225141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.225848 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.225849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:52.226699 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:52.226702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.227416 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.227418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:52.227500 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.227504 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:22:52.227556 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 08:22:52.227620 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":2,"thread_start_us":63,"threads_started":1}
I20260619 08:22:52.227766 22218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.224667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:33729 (local address 127.4.151.129:59229)
0619 08:22:52.224822 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:52.224832 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:52.224895 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:52.225080 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:52.225081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:52.225083 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:52.225164 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:52.225165 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.225890 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.225891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:52.226780 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:52.226783 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.227455 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.227456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:52.227527 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.227530 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:22:52.227606 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 08:22:52.227647 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
I20260619 08:22:52.228109 22011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7cb438427c440d996e9e50f2ee9834" candidate_uuid: "f6a13f10e05b49579e21acf15ddab6bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca97332580e949128666405b87d6ff4f" is_pre_election: true
I20260619 08:22:52.228102 22145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7cb438427c440d996e9e50f2ee9834" candidate_uuid: "f6a13f10e05b49579e21acf15ddab6bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15482df87c4a4487bbb49c9728dbbd4c" is_pre_election: true
I20260619 08:22:52.228269 22145 raft_consensus.cc:2454] T 4a7cb438427c440d996e9e50f2ee9834 P 15482df87c4a4487bbb49c9728dbbd4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6a13f10e05b49579e21acf15ddab6bc in term 0.
I20260619 08:22:52.228269 22011 raft_consensus.cc:2454] T 4a7cb438427c440d996e9e50f2ee9834 P ca97332580e949128666405b87d6ff4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6a13f10e05b49579e21acf15ddab6bc in term 0.
I20260619 08:22:52.228505 21813 leader_election.cc:302] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15482df87c4a4487bbb49c9728dbbd4c, f6a13f10e05b49579e21acf15ddab6bc; no voters:
I20260619 08:22:52.228662 22213 raft_consensus.cc:2781] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:22:52.228719 22213 raft_consensus.cc:482] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:22:52.228744 22213 raft_consensus.cc:3037] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:52.229487 22213 raft_consensus.cc:504] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97332580e949128666405b87d6ff4f" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37191 } } peers { permanent_uuid: "f6a13f10e05b49579e21acf15ddab6bc" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 39335 } } peers { permanent_uuid: "15482df87c4a4487bbb49c9728dbbd4c" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33729 } }
I20260619 08:22:52.229640 22213 leader_election.cc:288] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc [CANDIDATE]: Term 1 election: Requested vote from peers ca97332580e949128666405b87d6ff4f (127.4.151.130:37191), 15482df87c4a4487bbb49c9728dbbd4c (127.4.151.131:33729)
I20260619 08:22:52.229799 22011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7cb438427c440d996e9e50f2ee9834" candidate_uuid: "f6a13f10e05b49579e21acf15ddab6bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca97332580e949128666405b87d6ff4f"
I20260619 08:22:52.229826 22145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7cb438427c440d996e9e50f2ee9834" candidate_uuid: "f6a13f10e05b49579e21acf15ddab6bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15482df87c4a4487bbb49c9728dbbd4c"
I20260619 08:22:52.229882 22011 raft_consensus.cc:3037] T 4a7cb438427c440d996e9e50f2ee9834 P ca97332580e949128666405b87d6ff4f [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:52.229948 22145 raft_consensus.cc:3037] T 4a7cb438427c440d996e9e50f2ee9834 P 15482df87c4a4487bbb49c9728dbbd4c [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:52.230759 22011 raft_consensus.cc:2454] T 4a7cb438427c440d996e9e50f2ee9834 P ca97332580e949128666405b87d6ff4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a13f10e05b49579e21acf15ddab6bc in term 1.
I20260619 08:22:52.230882 22145 raft_consensus.cc:2454] T 4a7cb438427c440d996e9e50f2ee9834 P 15482df87c4a4487bbb49c9728dbbd4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a13f10e05b49579e21acf15ddab6bc in term 1.
I20260619 08:22:52.230978 21811 leader_election.cc:302] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca97332580e949128666405b87d6ff4f, f6a13f10e05b49579e21acf15ddab6bc; no voters:
I20260619 08:22:52.231067 22213 raft_consensus.cc:2781] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:22:52.231225 22213 raft_consensus.cc:686] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc [term 1 LEADER]: Becoming Leader. State: Replica: f6a13f10e05b49579e21acf15ddab6bc, State: Running, Role: LEADER
I20260619 08:22:52.231345 22213 consensus_queue.cc:237] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97332580e949128666405b87d6ff4f" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37191 } } peers { permanent_uuid: "f6a13f10e05b49579e21acf15ddab6bc" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 39335 } } peers { permanent_uuid: "15482df87c4a4487bbb49c9728dbbd4c" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33729 } }
I20260619 08:22:52.232262 21731 catalog_manager.cc:5495] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc reported cstate change: term changed from 0 to 1, leader changed from <none> to f6a13f10e05b49579e21acf15ddab6bc (127.4.151.129). New cstate: current_term: 1 leader_uuid: "f6a13f10e05b49579e21acf15ddab6bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97332580e949128666405b87d6ff4f" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a13f10e05b49579e21acf15ddab6bc" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 39335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15482df87c4a4487bbb49c9728dbbd4c" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33729 } health_report { overall_health: UNKNOWN } } }
W20260619 08:22:52.263098 22192 tablet.cc:2369] T 4a7cb438427c440d996e9e50f2ee9834 P 15482df87c4a4487bbb49c9728dbbd4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:22:52.277055 22207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.274406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56036 (local address 127.4.151.129:39335)
0619 08:22:52.274489 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:52.274491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:52.274499 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:22:52.274551 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:52.274552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:52.274569 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:52.274642 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:22:52.274790 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.275346 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.276171 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.276384 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.276449 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.276601 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:22:52.276713 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:22:52.276851 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:52.276895 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 08:22:52.276926 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 08:22:52.279070 22145 raft_consensus.cc:1261] T 4a7cb438427c440d996e9e50f2ee9834 P 15482df87c4a4487bbb49c9728dbbd4c [term 1 FOLLOWER]: Refusing update from remote peer f6a13f10e05b49579e21acf15ddab6bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:22:52.279073 22011 raft_consensus.cc:1261] T 4a7cb438427c440d996e9e50f2ee9834 P ca97332580e949128666405b87d6ff4f [term 1 FOLLOWER]: Refusing update from remote peer f6a13f10e05b49579e21acf15ddab6bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:22:52.279474 22213 consensus_queue.cc:1035] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca97332580e949128666405b87d6ff4f" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:22:52.279609 22219 consensus_queue.cc:1035] T 4a7cb438427c440d996e9e50f2ee9834 P f6a13f10e05b49579e21acf15ddab6bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "15482df87c4a4487bbb49c9728dbbd4c" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 33729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:22:52.282919 22224 mvcc.cc:205] Tried to move back new op lower bound from 7298487796852379648 to 7298487796660228096. Current Snapshot: MvccSnapshot[applied={T|T < 7298487796852379648}]
I20260619 08:22:52.284675 21732 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:22:52.284785 21732 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:22:52.285670 21732 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89b0303685174b7b9dabef521fce1063: Sending DeleteTablet for 3 replicas of tablet 4a7cb438427c440d996e9e50f2ee9834
I20260619 08:22:52.286051 22125 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a7cb438427c440d996e9e50f2ee9834 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:22:52 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:34972
I20260619 08:22:52.286095 21857 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a7cb438427c440d996e9e50f2ee9834 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:22:52 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:56022
I20260619 08:22:52.286291 21991 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a7cb438427c440d996e9e50f2ee9834 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:22:52 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:35906
I20260619 08:22:52.286312 22231 tablet_replica.cc:330] T 4a7cb438427c440d996e9e50f2ee9834 P 15482df87c4a4487bbb49c9728dbbd4c: stopping tablet replica
I20260619 08:22:52.286339 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 21794
I20260619 08:22:52.286418 22231 raft_consensus.cc:2229] T 4a7cb438427c440d996e9e50f2ee9834 P 15482df87c4a4487bbb49c9728dbbd4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:22:52.286526 22231 raft_consensus.cc:2258] T 4a7cb438427c440d996e9e50f2ee9834 P 15482df87c4a4487bbb49c9728dbbd4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:22:52.286963 22233 tablet_replica.cc:330] T 4a7cb438427c440d996e9e50f2ee9834 P ca97332580e949128666405b87d6ff4f: stopping tablet replica
I20260619 08:22:52.287081 22233 raft_consensus.cc:2229] T 4a7cb438427c440d996e9e50f2ee9834 P ca97332580e949128666405b87d6ff4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:22:52.287204 22233 raft_consensus.cc:2258] T 4a7cb438427c440d996e9e50f2ee9834 P ca97332580e949128666405b87d6ff4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:22:52.287714 22233 ts_tablet_manager.cc:1905] T 4a7cb438427c440d996e9e50f2ee9834 P ca97332580e949128666405b87d6ff4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:22:52.287976 22231 ts_tablet_manager.cc:1905] T 4a7cb438427c440d996e9e50f2ee9834 P 15482df87c4a4487bbb49c9728dbbd4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:22:52.289515 22233 ts_tablet_manager.cc:1918] T 4a7cb438427c440d996e9e50f2ee9834 P ca97332580e949128666405b87d6ff4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:22:52.289589 22233 log.cc:1196] T 4a7cb438427c440d996e9e50f2ee9834 P ca97332580e949128666405b87d6ff4f: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/4a7cb438427c440d996e9e50f2ee9834
I20260619 08:22:52.289892 22233 ts_tablet_manager.cc:1939] T 4a7cb438427c440d996e9e50f2ee9834 P ca97332580e949128666405b87d6ff4f: Deleting consensus metadata
I20260619 08:22:52.290012 22231 ts_tablet_manager.cc:1918] T 4a7cb438427c440d996e9e50f2ee9834 P 15482df87c4a4487bbb49c9728dbbd4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:22:52.290123 22231 log.cc:1196] T 4a7cb438427c440d996e9e50f2ee9834 P 15482df87c4a4487bbb49c9728dbbd4c: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/4a7cb438427c440d996e9e50f2ee9834
I20260619 08:22:52.290406 22231 ts_tablet_manager.cc:1939] T 4a7cb438427c440d996e9e50f2ee9834 P 15482df87c4a4487bbb49c9728dbbd4c: Deleting consensus metadata
I20260619 08:22:52.290453 21717 catalog_manager.cc:4841] TS ca97332580e949128666405b87d6ff4f (127.4.151.130:37191): tablet 4a7cb438427c440d996e9e50f2ee9834 (table test-table [id=1a3a0c21440a4c358a83c58c4e70df46]) successfully deleted
I20260619 08:22:52.290863 21719 catalog_manager.cc:4841] TS 15482df87c4a4487bbb49c9728dbbd4c (127.4.151.131:33729): tablet 4a7cb438427c440d996e9e50f2ee9834 (table test-table [id=1a3a0c21440a4c358a83c58c4e70df46]) successfully deleted
W20260619 08:22:52.293474 21716 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.151.129:39335 (error 108)
I20260619 08:22:52.293515 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 21928
I20260619 08:22:52.295140 22203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.294986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:39335 (local address 127.0.0.1:56042)
0619 08:22:52.295050 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:52.295095 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.129:39335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 08:22:52.295420 21716 catalog_manager.cc:4584] TS f6a13f10e05b49579e21acf15ddab6bc (127.4.151.129:39335): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4a7cb438427c440d996e9e50f2ee9834: Network error: Client connection negotiation failed: client connection to 127.4.151.129:39335: connect: Connection refused (error 111)
I20260619 08:22:52.299145 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 22062
I20260619 08:22:52.303843 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 21700
2026-06-19T08:22:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2161 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T08:22:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:22:52Z Disabled control of system clock
I20260619 08:22:52.332248 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:39027
--webserver_interface=127.4.151.190
--webserver_port=0
--builtin_ntp_servers=127.4.151.148:38853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:39027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 08:22:52.415264 22238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:52.415495 22238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:52.415517 22238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:52.416877 22238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:22:52.416923 22238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:52.416936 22238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:52.416949 22238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:22:52.416962 22238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:22:52.418481 22238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:38853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:39027
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:39027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:52.418702 22238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:52.418951 22238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:22:52.424181 22238 server_base.cc:1034] running on GCE node
W20260619 08:22:52.424149 22243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:52.424146 22244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:52.424149 22246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:52.424793 22238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:52.425092 22238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:52.426343 22238 hybrid_clock.cc:648] HybridClock initialized: now 1781857372426337 us; error 39 us; skew 500 ppm
I20260619 08:22:52.427601 22238 webserver.cc:460] Webserver started at http://127.4.151.190:34687/ using document root <none> and password file <none>
I20260619 08:22:52.427845 22238 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:52.427917 22238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:52.428084 22238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:22:52.429230 22238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "27cc149d49db44848225225b33f8ec25"
format_stamp: "Formatted at 2026-06-19 08:22:52 on dist-test-slave-d3n8"
server_key: "54e0618dd1c630febddee2e9700cd9fa"
server_key_iv: "a00e4e1b27b9f1cb1415b83037d299e6"
server_key_version: "encryptionkey@0"
I20260619 08:22:52.429628 22238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "27cc149d49db44848225225b33f8ec25"
format_stamp: "Formatted at 2026-06-19 08:22:52 on dist-test-slave-d3n8"
server_key: "54e0618dd1c630febddee2e9700cd9fa"
server_key_iv: "a00e4e1b27b9f1cb1415b83037d299e6"
server_key_version: "encryptionkey@0"
I20260619 08:22:52.430976 22238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:22:52.431746 22252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:52.431916 22238 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:22:52.432015 22238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "27cc149d49db44848225225b33f8ec25"
format_stamp: "Formatted at 2026-06-19 08:22:52 on dist-test-slave-d3n8"
server_key: "54e0618dd1c630febddee2e9700cd9fa"
server_key_iv: "a00e4e1b27b9f1cb1415b83037d299e6"
server_key_version: "encryptionkey@0"
I20260619 08:22:52.432106 22238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:52.444890 22238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:52.445199 22238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:52.445353 22238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:52.449050 22238 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:39027
I20260619 08:22:52.449137 22304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:39027 every 8 connection(s)
I20260619 08:22:52.449523 22238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:22:52.450563 22305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:52.452780 22305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27cc149d49db44848225225b33f8ec25: Bootstrap starting.
I20260619 08:22:52.453307 22305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27cc149d49db44848225225b33f8ec25: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:52.453644 22305 log.cc:824] T 00000000000000000000000000000000 P 27cc149d49db44848225225b33f8ec25: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:52.454331 22305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27cc149d49db44848225225b33f8ec25: No bootstrap required, opened a new log
I20260619 08:22:52.455847 22305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27cc149d49db44848225225b33f8ec25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27cc149d49db44848225225b33f8ec25" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 39027 } }
I20260619 08:22:52.455982 22305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27cc149d49db44848225225b33f8ec25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:52.456004 22305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27cc149d49db44848225225b33f8ec25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27cc149d49db44848225225b33f8ec25, State: Initialized, Role: FOLLOWER
I20260619 08:22:52.456143 22305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27cc149d49db44848225225b33f8ec25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27cc149d49db44848225225b33f8ec25" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 39027 } }
I20260619 08:22:52.456226 22305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27cc149d49db44848225225b33f8ec25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:22:52.456249 22305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27cc149d49db44848225225b33f8ec25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:22:52.456280 22305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27cc149d49db44848225225b33f8ec25 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:52.456748 22305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27cc149d49db44848225225b33f8ec25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27cc149d49db44848225225b33f8ec25" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 39027 } }
I20260619 08:22:52.456840 22305 leader_election.cc:302] T 00000000000000000000000000000000 P 27cc149d49db44848225225b33f8ec25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27cc149d49db44848225225b33f8ec25; no voters:
I20260619 08:22:52.457067 22305 leader_election.cc:288] T 00000000000000000000000000000000 P 27cc149d49db44848225225b33f8ec25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:22:52.457154 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 22238
I20260619 08:22:52.457171 22308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27cc149d49db44848225225b33f8ec25 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:22:52.457275 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:22:52.457340 22308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27cc149d49db44848225225b33f8ec25 [term 1 LEADER]: Becoming Leader. State: Replica: 27cc149d49db44848225225b33f8ec25, State: Running, Role: LEADER
I20260619 08:22:52.457441 22305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27cc149d49db44848225225b33f8ec25 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:22:52.457508 4702 external_mini_cluster.cc:1428] Setting key 7eca4ba7fbec1ad497f4c8c35a26f3d0
I20260619 08:22:52.457479 22308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27cc149d49db44848225225b33f8ec25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27cc149d49db44848225225b33f8ec25" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 39027 } }
I20260619 08:22:52.457819 22308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27cc149d49db44848225225b33f8ec25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27cc149d49db44848225225b33f8ec25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27cc149d49db44848225225b33f8ec25" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 39027 } } }
I20260619 08:22:52.457909 22308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27cc149d49db44848225225b33f8ec25 [sys.catalog]: This master's current role is: LEADER
I20260619 08:22:52.458051 22308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27cc149d49db44848225225b33f8ec25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27cc149d49db44848225225b33f8ec25. Latest consensus state: current_term: 1 leader_uuid: "27cc149d49db44848225225b33f8ec25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27cc149d49db44848225225b33f8ec25" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 39027 } } }
I20260619 08:22:52.458166 22308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27cc149d49db44848225225b33f8ec25 [sys.catalog]: This master's current role is: LEADER
I20260619 08:22:52.458335 22315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:22:52.459134 22315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:22:52.461027 22315 catalog_manager.cc:1347] Generated new cluster ID: c7b1ebb7b252490e9f5bbd6c76e3d1c1
I20260619 08:22:52.461087 22315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:22:52.464375 22319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.458184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58936 (local address 127.4.151.190:39027)
0619 08:22:52.458398 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:52.458408 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:52.458439 (+ 31us) server_negotiation.cc:407] Connection header received
0619 08:22:52.458556 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:52.458563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:52.458646 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:52.459027 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:52.459141 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.460525 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.461693 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.462437 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.462944 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.463466 (+ 522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:52.463471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:52.463471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:22:52.463705 (+ 234us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:52.463749 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:22:52.463767 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:52.464014 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:52.464049 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 08:22:52.464143 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
I20260619 08:22:52.467805 22315 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:22:52.468484 22315 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:22:52.477521 22315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27cc149d49db44848225225b33f8ec25: Generated new TSK 0
I20260619 08:22:52.477747 22315 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:22:52.483229 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:39027
--builtin_ntp_servers=127.4.151.148:38853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 08:22:52.567919 22329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:52.568120 22329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:52.568141 22329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:52.569348 22329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:52.569444 22329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:52.569485 22329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:22:52.570907 22329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:38853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:39027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:52.571156 22329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:52.571487 22329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:22:52.577762 22329 server_base.cc:1034] running on GCE node
W20260619 08:22:52.577731 22335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:52.577740 22337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:52.577965 22334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:52.578176 22329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:52.578439 22329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:52.579612 22329 hybrid_clock.cc:648] HybridClock initialized: now 1781857372579591 us; error 37 us; skew 500 ppm
I20260619 08:22:52.580969 22329 webserver.cc:460] Webserver started at http://127.4.151.129:36627/ using document root <none> and password file <none>
I20260619 08:22:52.581298 22329 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:52.581380 22329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:52.581568 22329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:22:52.582656 22329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "77e297613f294ea9a9e0bf4acf95f1b8"
format_stamp: "Formatted at 2026-06-19 08:22:52 on dist-test-slave-d3n8"
server_key: "4c5235b8ccbeeae8f19ad5baf2e2a1c2"
server_key_iv: "1bb695e6af1e868011e0df9937502527"
server_key_version: "encryptionkey@0"
I20260619 08:22:52.583024 22329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "77e297613f294ea9a9e0bf4acf95f1b8"
format_stamp: "Formatted at 2026-06-19 08:22:52 on dist-test-slave-d3n8"
server_key: "4c5235b8ccbeeae8f19ad5baf2e2a1c2"
server_key_iv: "1bb695e6af1e868011e0df9937502527"
server_key_version: "encryptionkey@0"
I20260619 08:22:52.584420 22329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:22:52.585204 22343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:52.585409 22329 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:22:52.585521 22329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "77e297613f294ea9a9e0bf4acf95f1b8"
format_stamp: "Formatted at 2026-06-19 08:22:52 on dist-test-slave-d3n8"
server_key: "4c5235b8ccbeeae8f19ad5baf2e2a1c2"
server_key_iv: "1bb695e6af1e868011e0df9937502527"
server_key_version: "encryptionkey@0"
I20260619 08:22:52.585595 22329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:52.600834 22329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:52.601167 22329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:52.601330 22329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:52.601708 22329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:22:52.602090 22329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:22:52.602129 22329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:52.602154 22329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:22:52.602207 22329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:52.607862 22329 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:38055
I20260619 08:22:52.607940 22456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:38055 every 8 connection(s)
I20260619 08:22:52.608290 22329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
I20260619 08:22:52.608587 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 22329
I20260619 08:22:52.608703 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:22:52.608942 4702 external_mini_cluster.cc:1428] Setting key 66781f92e694c0c2dbb0ff90d8c88be8
I20260619 08:22:52.610054 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:39027
--builtin_ntp_servers=127.4.151.148:38853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 08:22:52.614181 22319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.609311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:52849 (local address 127.4.151.190:39027)
0619 08:22:52.609411 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:52.609414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:52.609533 (+ 119us) server_negotiation.cc:407] Connection header received
0619 08:22:52.610147 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:52.610148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:52.610163 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:52.610233 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:52.610962 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.611593 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.612659 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.612864 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.612953 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.613883 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:52.613884 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:52.613885 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:22:52.613890 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:52.613906 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:22:52.613910 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:52.613954 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:52.614065 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 08:22:52.614100 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 08:22:52.614483 22459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.609148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:39027 (local address 127.4.151.129:52849)
0619 08:22:52.609357 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:52.609431 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:52.610018 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:52.610311 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:52.610315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:52.610325 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:22:52.610864 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:52.610868 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.611704 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.611705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:52.612550 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:52.612552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.612966 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.612968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:52.613106 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.613741 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:22:52.613742 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:22:52.613771 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:22:52.613793 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:22:52.613964 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:22:52.613969 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:22:52.613972 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:22:52.614068 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 08:22:52.614171 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260619 08:22:52.614992 22457 heartbeater.cc:347] Connected to a master server at 127.4.151.190:39027
I20260619 08:22:52.615106 22457 heartbeater.cc:464] Registering TS with master...
I20260619 08:22:52.615341 22457 heartbeater.cc:511] Master 127.4.151.190:39027 requested a full tablet report, sending...
I20260619 08:22:52.615954 22269 ts_manager.cc:194] Registered new tserver with Master: 77e297613f294ea9a9e0bf4acf95f1b8 (127.4.151.129:38055)
I20260619 08:22:52.616642 22269 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.151.129:52849
W20260619 08:22:52.693929 22460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:52.694165 22460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:52.694188 22460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:52.695420 22460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:52.695472 22460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:52.695497 22460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:22:52.696928 22460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:38853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:39027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:52.697207 22460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:52.697575 22460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:52.703230 22465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:52.703291 22468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:52.703291 22466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:52.703496 22460 server_base.cc:1034] running on GCE node
I20260619 08:22:52.703738 22460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:52.703982 22460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:52.705152 22460 hybrid_clock.cc:648] HybridClock initialized: now 1781857372705104 us; error 73 us; skew 500 ppm
I20260619 08:22:52.706638 22460 webserver.cc:460] Webserver started at http://127.4.151.130:34277/ using document root <none> and password file <none>
I20260619 08:22:52.706895 22460 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:52.706940 22460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:52.707034 22460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:22:52.708077 22460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "109aa9cc9392474fa1733c48c96c1d35"
format_stamp: "Formatted at 2026-06-19 08:22:52 on dist-test-slave-d3n8"
server_key: "eb670a3ec5dec92d70aa46d1877db999"
server_key_iv: "d4ae9bea3f51fe45524316a87feb728e"
server_key_version: "encryptionkey@0"
I20260619 08:22:52.708375 22460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "109aa9cc9392474fa1733c48c96c1d35"
format_stamp: "Formatted at 2026-06-19 08:22:52 on dist-test-slave-d3n8"
server_key: "eb670a3ec5dec92d70aa46d1877db999"
server_key_iv: "d4ae9bea3f51fe45524316a87feb728e"
server_key_version: "encryptionkey@0"
I20260619 08:22:52.709726 22460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:22:52.710584 22474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:52.710839 22460 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:22:52.710958 22460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "109aa9cc9392474fa1733c48c96c1d35"
format_stamp: "Formatted at 2026-06-19 08:22:52 on dist-test-slave-d3n8"
server_key: "eb670a3ec5dec92d70aa46d1877db999"
server_key_iv: "d4ae9bea3f51fe45524316a87feb728e"
server_key_version: "encryptionkey@0"
I20260619 08:22:52.711038 22460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:52.719745 22460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:52.720062 22460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:52.720211 22460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:52.720454 22460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:22:52.720778 22460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:22:52.720813 22460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:52.720835 22460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:22:52.720881 22460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:52.726625 22460 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:37867
I20260619 08:22:52.726699 22587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:37867 every 8 connection(s)
I20260619 08:22:52.727043 22460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
I20260619 08:22:52.732486 22319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.727809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:39911 (local address 127.4.151.190:39027)
0619 08:22:52.727880 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:52.727882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:52.728124 (+ 242us) server_negotiation.cc:407] Connection header received
0619 08:22:52.728665 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:52.728666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:52.728679 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:52.728730 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:52.729479 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.729986 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.730985 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.731183 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.731270 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.732166 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:52.732167 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:52.732168 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:22:52.732173 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:52.732188 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:22:52.732193 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:52.732269 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:52.732371 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 08:22:52.732401 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 08:22:52.732729 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.727805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:39027 (local address 127.4.151.130:39911)
0619 08:22:52.727996 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:52.728031 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:52.728539 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:52.728804 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:52.728808 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:52.728815 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:22:52.729299 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:52.729303 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.730084 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.730085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:52.730877 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:52.730879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.731281 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.731283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:52.731419 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.732006 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:22:52.732006 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:22:52.732034 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:22:52.732059 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:22:52.732271 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:22:52.732277 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:22:52.732280 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:22:52.732369 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 08:22:52.732473 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
I20260619 08:22:52.733204 22588 heartbeater.cc:347] Connected to a master server at 127.4.151.190:39027
I20260619 08:22:52.733299 22588 heartbeater.cc:464] Registering TS with master...
I20260619 08:22:52.733517 22588 heartbeater.cc:511] Master 127.4.151.190:39027 requested a full tablet report, sending...
I20260619 08:22:52.733927 22269 ts_manager.cc:194] Registered new tserver with Master: 109aa9cc9392474fa1733c48c96c1d35 (127.4.151.130:37867)
I20260619 08:22:52.734550 22269 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.151.130:39911
I20260619 08:22:52.734997 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 22460
I20260619 08:22:52.735141 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:22:52.735359 4702 external_mini_cluster.cc:1428] Setting key c14d2014eff4e3075a806cfbad5793b3
I20260619 08:22:52.736286 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:39027
--builtin_ntp_servers=127.4.151.148:38853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 08:22:52.818825 22591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:52.819080 22591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:52.819116 22591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:52.820310 22591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:52.820384 22591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:52.820429 22591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:22:52.821938 22591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:38853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:39027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:52.822207 22591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:52.822515 22591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:52.827853 22597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:52.827901 22591 server_base.cc:1034] running on GCE node
W20260619 08:22:52.827864 22599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:52.827884 22596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:52.828339 22591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:52.828625 22591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:52.829795 22591 hybrid_clock.cc:648] HybridClock initialized: now 1781857372829771 us; error 37 us; skew 500 ppm
I20260619 08:22:52.831295 22591 webserver.cc:460] Webserver started at http://127.4.151.131:33159/ using document root <none> and password file <none>
I20260619 08:22:52.831542 22591 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:52.831610 22591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:52.831749 22591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:22:52.832891 22591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "f867f38697ee4638828375f88dc42eaa"
format_stamp: "Formatted at 2026-06-19 08:22:52 on dist-test-slave-d3n8"
server_key: "dd043ea1c7904a09cf6a7dfdaca0103d"
server_key_iv: "9f37951729c870ef9e59785a1d826405"
server_key_version: "encryptionkey@0"
I20260619 08:22:52.833278 22591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "f867f38697ee4638828375f88dc42eaa"
format_stamp: "Formatted at 2026-06-19 08:22:52 on dist-test-slave-d3n8"
server_key: "dd043ea1c7904a09cf6a7dfdaca0103d"
server_key_iv: "9f37951729c870ef9e59785a1d826405"
server_key_version: "encryptionkey@0"
I20260619 08:22:52.834830 22591 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:22:52.835803 22605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:52.836037 22591 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:22:52.836136 22591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "f867f38697ee4638828375f88dc42eaa"
format_stamp: "Formatted at 2026-06-19 08:22:52 on dist-test-slave-d3n8"
server_key: "dd043ea1c7904a09cf6a7dfdaca0103d"
server_key_iv: "9f37951729c870ef9e59785a1d826405"
server_key_version: "encryptionkey@0"
I20260619 08:22:52.836285 22591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:52.858451 22591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:52.858801 22591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:52.858968 22591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:52.859226 22591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:22:52.859560 22591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:22:52.859615 22591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:52.859658 22591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:22:52.859702 22591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:52.865506 22591 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:45669
I20260619 08:22:52.865553 22718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:45669 every 8 connection(s)
I20260619 08:22:52.865947 22591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
I20260619 08:22:52.871723 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 22591
I20260619 08:22:52.871768 22319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.866848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:51493 (local address 127.4.151.190:39027)
0619 08:22:52.866923 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:52.866929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:52.867080 (+ 151us) server_negotiation.cc:407] Connection header received
0619 08:22:52.867599 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:52.867601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:52.867626 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:52.867695 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:52.868477 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.869028 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.870175 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.870392 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.870461 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.871453 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:52.871454 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:52.871454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:22:52.871459 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:52.871474 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:22:52.871478 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:52.871533 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:52.871673 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 08:22:52.871704 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 08:22:52.871829 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:22:52.872014 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.866760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:39027 (local address 127.4.151.131:51493)
0619 08:22:52.866945 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:52.866986 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:52.867472 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:52.867772 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:52.867776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:52.867785 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:22:52.868361 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:52.868365 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.869125 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.869126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:52.870066 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:52.870068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.870493 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.870493 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:52.870613 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.871277 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:22:52.871277 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:22:52.871307 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:22:52.871329 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:22:52.871550 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:22:52.871556 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:22:52.871559 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:22:52.871638 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 08:22:52.871760 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260619 08:22:52.872035 4702 external_mini_cluster.cc:1428] Setting key f72e148bedba6023e54057d7868a3a17
I20260619 08:22:52.872704 22719 heartbeater.cc:347] Connected to a master server at 127.4.151.190:39027
I20260619 08:22:52.872821 22719 heartbeater.cc:464] Registering TS with master...
I20260619 08:22:52.873024 22719 heartbeater.cc:511] Master 127.4.151.190:39027 requested a full tablet report, sending...
I20260619 08:22:52.873483 22269 ts_manager.cc:194] Registered new tserver with Master: f867f38697ee4638828375f88dc42eaa (127.4.151.131:45669)
I20260619 08:22:52.873874 22269 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.151.131:51493
I20260619 08:22:52.884371 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:22:52.889251 22319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.886058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58946 (local address 127.4.151.190:39027)
0619 08:22:52.886196 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:52.886198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:52.886208 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:22:52.886250 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:52.886251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:52.886263 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:52.886324 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:52.886497 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.887002 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.887950 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.888148 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.888220 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.889019 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:52.889021 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:52.889021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:22:52.889025 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:52.889041 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:22:52.889045 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:52.889096 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:52.889150 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 08:22:52.889180 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260619 08:22:52.891316 22269 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:22:52.893565 22269 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:22:52.900847 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.896269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:38055 (local address 127.0.0.1:60322)
0619 08:22:52.896424 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:52.896434 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:52.896484 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:52.897165 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:52.897166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:52.897169 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:52.897208 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:52.897209 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.898512 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.898513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:52.899564 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:52.899567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.900511 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.900512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:52.900604 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.900622 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 08:22:52.900689 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 08:22:52.900756 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
I20260619 08:22:52.900910 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.896115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38356 (local address 127.4.151.130:37867)
0619 08:22:52.896392 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:52.896398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:52.896410 (+ 12us) server_negotiation.cc:407] Connection header received
0619 08:22:52.896522 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:52.896524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:52.896542 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:52.896904 (+ 362us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:52.896973 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.898165 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.899978 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.900535 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.900631 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.900729 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:52.900783 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 08:22:52.900821 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":92,"threads_started":1}
I20260619 08:22:52.901000 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.896818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43274 (local address 127.4.151.131:45669)
0619 08:22:52.897083 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:52.897088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:52.897097 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:22:52.897131 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:52.897133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:52.897148 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:52.897218 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:52.897468 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.898209 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.899739 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.900441 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.900502 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.900595 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:52.900864 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 08:22:52.900910 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":102,"threads_started":1}
I20260619 08:22:52.901186 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.896740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60322 (local address 127.4.151.129:38055)
0619 08:22:52.896890 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:52.896895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:52.896906 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:22:52.896944 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:52.896946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:52.896961 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:52.897044 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:52.897292 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.898445 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.899659 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.900408 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.900497 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.900595 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:52.901062 (+ 467us) server_negotiation.cc:299] Negotiation successful
0619 08:22:52.901106 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260619 08:22:52.901214 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.895971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:37867 (local address 127.0.0.1:38356)
0619 08:22:52.896231 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:52.896330 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:52.896414 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:52.896662 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:52.896664 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:52.896667 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:52.896752 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:52.896754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.898282 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.898283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:52.899882 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:52.899884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.900642 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.900642 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:52.900702 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.900709 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 08:22:52.900761 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 08:22:52.900818 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":79,"thread_start_us":134,"threads_started":1}
I20260619 08:22:52.901214 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:52.896471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:45669 (local address 127.0.0.1:43274)
0619 08:22:52.896586 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:52.896596 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:52.896638 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:52.897302 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:52.897303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:52.897305 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:52.897350 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:52.897352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.898304 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.898305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:52.899654 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:52.899657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:52.900634 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:52.900635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:52.900702 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:52.900709 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 08:22:52.900760 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 08:22:52.900818 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":78,"thread_start_us":65,"threads_started":1}
I20260619 08:22:52.902184 22653 tablet_service.cc:1467] Processing CreateTablet for tablet c73cdec703c34f569e22678ec33b6523 (DEFAULT_TABLE table=test-table [id=08bbc28b045e4c09bc649caffa67558d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:22:52.902184 22391 tablet_service.cc:1467] Processing CreateTablet for tablet c73cdec703c34f569e22678ec33b6523 (DEFAULT_TABLE table=test-table [id=08bbc28b045e4c09bc649caffa67558d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:22:52.902536 22391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c73cdec703c34f569e22678ec33b6523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:52.902539 22653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c73cdec703c34f569e22678ec33b6523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:52.905332 22733 tablet_bootstrap.cc:492] T c73cdec703c34f569e22678ec33b6523 P 77e297613f294ea9a9e0bf4acf95f1b8: Bootstrap starting.
I20260619 08:22:52.905558 22734 tablet_bootstrap.cc:492] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa: Bootstrap starting.
I20260619 08:22:52.905630 22522 tablet_service.cc:1467] Processing CreateTablet for tablet c73cdec703c34f569e22678ec33b6523 (DEFAULT_TABLE table=test-table [id=08bbc28b045e4c09bc649caffa67558d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:22:52.905978 22522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c73cdec703c34f569e22678ec33b6523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:52.906121 22734 tablet_bootstrap.cc:654] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:52.906121 22733 tablet_bootstrap.cc:654] T c73cdec703c34f569e22678ec33b6523 P 77e297613f294ea9a9e0bf4acf95f1b8: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:52.906466 22733 log.cc:824] T c73cdec703c34f569e22678ec33b6523 P 77e297613f294ea9a9e0bf4acf95f1b8: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:52.906466 22734 log.cc:824] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:52.907244 22734 tablet_bootstrap.cc:492] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa: No bootstrap required, opened a new log
I20260619 08:22:52.907254 22733 tablet_bootstrap.cc:492] T c73cdec703c34f569e22678ec33b6523 P 77e297613f294ea9a9e0bf4acf95f1b8: No bootstrap required, opened a new log
I20260619 08:22:52.907318 22733 ts_tablet_manager.cc:1397] T c73cdec703c34f569e22678ec33b6523 P 77e297613f294ea9a9e0bf4acf95f1b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:22:52.907318 22734 ts_tablet_manager.cc:1397] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:22:52.908962 22737 tablet_bootstrap.cc:492] T c73cdec703c34f569e22678ec33b6523 P 109aa9cc9392474fa1733c48c96c1d35: Bootstrap starting.
I20260619 08:22:52.909623 22737 tablet_bootstrap.cc:654] T c73cdec703c34f569e22678ec33b6523 P 109aa9cc9392474fa1733c48c96c1d35: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:52.909929 22737 log.cc:824] T c73cdec703c34f569e22678ec33b6523 P 109aa9cc9392474fa1733c48c96c1d35: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:52.909950 22733 raft_consensus.cc:348] T c73cdec703c34f569e22678ec33b6523 P 77e297613f294ea9a9e0bf4acf95f1b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109aa9cc9392474fa1733c48c96c1d35" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37867 } } peers { permanent_uuid: "77e297613f294ea9a9e0bf4acf95f1b8" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 38055 } } peers { permanent_uuid: "f867f38697ee4638828375f88dc42eaa" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 45669 } }
I20260619 08:22:52.909950 22734 raft_consensus.cc:348] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109aa9cc9392474fa1733c48c96c1d35" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37867 } } peers { permanent_uuid: "77e297613f294ea9a9e0bf4acf95f1b8" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 38055 } } peers { permanent_uuid: "f867f38697ee4638828375f88dc42eaa" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 45669 } }
I20260619 08:22:52.910126 22734 raft_consensus.cc:374] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:52.910126 22733 raft_consensus.cc:374] T c73cdec703c34f569e22678ec33b6523 P 77e297613f294ea9a9e0bf4acf95f1b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:52.910162 22733 raft_consensus.cc:729] T c73cdec703c34f569e22678ec33b6523 P 77e297613f294ea9a9e0bf4acf95f1b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e297613f294ea9a9e0bf4acf95f1b8, State: Initialized, Role: FOLLOWER
I20260619 08:22:52.910162 22734 raft_consensus.cc:729] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f867f38697ee4638828375f88dc42eaa, State: Initialized, Role: FOLLOWER
I20260619 08:22:52.910293 22733 consensus_queue.cc:260] T c73cdec703c34f569e22678ec33b6523 P 77e297613f294ea9a9e0bf4acf95f1b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109aa9cc9392474fa1733c48c96c1d35" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37867 } } peers { permanent_uuid: "77e297613f294ea9a9e0bf4acf95f1b8" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 38055 } } peers { permanent_uuid: "f867f38697ee4638828375f88dc42eaa" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 45669 } }
I20260619 08:22:52.910293 22734 consensus_queue.cc:260] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109aa9cc9392474fa1733c48c96c1d35" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37867 } } peers { permanent_uuid: "77e297613f294ea9a9e0bf4acf95f1b8" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 38055 } } peers { permanent_uuid: "f867f38697ee4638828375f88dc42eaa" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 45669 } }
I20260619 08:22:52.910557 22733 ts_tablet_manager.cc:1428] T c73cdec703c34f569e22678ec33b6523 P 77e297613f294ea9a9e0bf4acf95f1b8: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260619 08:22:52.910557 22734 ts_tablet_manager.cc:1428] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260619 08:22:52.910601 22719 heartbeater.cc:503] Master 127.4.151.190:39027 was elected leader, sending a full tablet report...
I20260619 08:22:52.910706 22457 heartbeater.cc:503] Master 127.4.151.190:39027 was elected leader, sending a full tablet report...
I20260619 08:22:52.911201 22737 tablet_bootstrap.cc:492] T c73cdec703c34f569e22678ec33b6523 P 109aa9cc9392474fa1733c48c96c1d35: No bootstrap required, opened a new log
I20260619 08:22:52.911278 22737 ts_tablet_manager.cc:1397] T c73cdec703c34f569e22678ec33b6523 P 109aa9cc9392474fa1733c48c96c1d35: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:22:52.913422 22737 raft_consensus.cc:348] T c73cdec703c34f569e22678ec33b6523 P 109aa9cc9392474fa1733c48c96c1d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109aa9cc9392474fa1733c48c96c1d35" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37867 } } peers { permanent_uuid: "77e297613f294ea9a9e0bf4acf95f1b8" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 38055 } } peers { permanent_uuid: "f867f38697ee4638828375f88dc42eaa" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 45669 } }
I20260619 08:22:52.913573 22737 raft_consensus.cc:374] T c73cdec703c34f569e22678ec33b6523 P 109aa9cc9392474fa1733c48c96c1d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:52.913595 22737 raft_consensus.cc:729] T c73cdec703c34f569e22678ec33b6523 P 109aa9cc9392474fa1733c48c96c1d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 109aa9cc9392474fa1733c48c96c1d35, State: Initialized, Role: FOLLOWER
I20260619 08:22:52.913733 22737 consensus_queue.cc:260] T c73cdec703c34f569e22678ec33b6523 P 109aa9cc9392474fa1733c48c96c1d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109aa9cc9392474fa1733c48c96c1d35" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37867 } } peers { permanent_uuid: "77e297613f294ea9a9e0bf4acf95f1b8" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 38055 } } peers { permanent_uuid: "f867f38697ee4638828375f88dc42eaa" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 45669 } }
I20260619 08:22:52.914000 22737 ts_tablet_manager.cc:1428] T c73cdec703c34f569e22678ec33b6523 P 109aa9cc9392474fa1733c48c96c1d35: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:22:52.914136 22588 heartbeater.cc:503] Master 127.4.151.190:39027 was elected leader, sending a full tablet report...
W20260619 08:22:52.977578 22589 tablet.cc:2369] T c73cdec703c34f569e22678ec33b6523 P 109aa9cc9392474fa1733c48c96c1d35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:22:53.066206 22740 raft_consensus.cc:482] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:22:53.066347 22740 raft_consensus.cc:504] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109aa9cc9392474fa1733c48c96c1d35" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37867 } } peers { permanent_uuid: "77e297613f294ea9a9e0bf4acf95f1b8" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 38055 } } peers { permanent_uuid: "f867f38697ee4638828375f88dc42eaa" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 45669 } }
I20260619 08:22:53.066763 22740 leader_election.cc:288] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 109aa9cc9392474fa1733c48c96c1d35 (127.4.151.130:37867), 77e297613f294ea9a9e0bf4acf95f1b8 (127.4.151.129:38055)
I20260619 08:22:53.069904 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:53.067040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:37867 (local address 127.4.151.131:53371)
0619 08:22:53.067129 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:53.067144 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:53.067217 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:53.067457 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:53.067458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:53.067459 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:53.067537 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:53.067539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:53.068221 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:53.068222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:53.068988 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:53.068990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:53.069629 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:53.069630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:53.069703 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:53.069707 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:22:53.069786 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 08:22:53.069831 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":1}
I20260619 08:22:53.069904 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:53.067268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:53371 (local address 127.4.151.130:37867)
0619 08:22:53.067317 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:53.067319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:53.067328 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:22:53.067360 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:53.067361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:53.067373 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:53.067450 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:53.067657 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:53.068134 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:53.069109 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:53.069534 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:53.069630 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:53.069639 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:53.069792 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 08:22:53.069831 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 08:22:53.070060 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:53.067247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:38055 (local address 127.4.151.131:51743)
0619 08:22:53.067446 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:53.067461 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:53.067532 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:53.067686 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:53.067687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:53.067689 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:53.067764 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:53.067765 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:53.068395 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:53.068396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:53.069154 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:53.069156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:53.069787 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:53.069788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:53.069873 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:53.069876 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:22:53.069940 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 08:22:53.069978 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":122,"threads_started":1}
I20260619 08:22:53.070065 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:53.067326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:51743 (local address 127.4.151.129:38055)
0619 08:22:53.067402 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:53.067404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:53.067522 (+ 118us) server_negotiation.cc:407] Connection header received
0619 08:22:53.067609 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:53.067610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:53.067620 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:53.067684 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:53.067838 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:53.068299 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:53.069254 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:53.069689 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:53.069784 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:53.069792 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:53.069959 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 08:22:53.070001 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 08:22:53.070358 22542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73cdec703c34f569e22678ec33b6523" candidate_uuid: "f867f38697ee4638828375f88dc42eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "109aa9cc9392474fa1733c48c96c1d35" is_pre_election: true
I20260619 08:22:53.070372 22411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73cdec703c34f569e22678ec33b6523" candidate_uuid: "f867f38697ee4638828375f88dc42eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e297613f294ea9a9e0bf4acf95f1b8" is_pre_election: true
I20260619 08:22:53.070528 22411 raft_consensus.cc:2454] T c73cdec703c34f569e22678ec33b6523 P 77e297613f294ea9a9e0bf4acf95f1b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f867f38697ee4638828375f88dc42eaa in term 0.
I20260619 08:22:53.070533 22542 raft_consensus.cc:2454] T c73cdec703c34f569e22678ec33b6523 P 109aa9cc9392474fa1733c48c96c1d35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f867f38697ee4638828375f88dc42eaa in term 0.
I20260619 08:22:53.070806 22607 leader_election.cc:302] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e297613f294ea9a9e0bf4acf95f1b8, f867f38697ee4638828375f88dc42eaa; no voters:
I20260619 08:22:53.070986 22740 raft_consensus.cc:2781] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:22:53.071074 22740 raft_consensus.cc:482] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:22:53.071092 22740 raft_consensus.cc:3037] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:53.071856 22740 raft_consensus.cc:504] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109aa9cc9392474fa1733c48c96c1d35" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37867 } } peers { permanent_uuid: "77e297613f294ea9a9e0bf4acf95f1b8" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 38055 } } peers { permanent_uuid: "f867f38697ee4638828375f88dc42eaa" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 45669 } }
I20260619 08:22:53.072014 22740 leader_election.cc:288] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa [CANDIDATE]: Term 1 election: Requested vote from peers 109aa9cc9392474fa1733c48c96c1d35 (127.4.151.130:37867), 77e297613f294ea9a9e0bf4acf95f1b8 (127.4.151.129:38055)
I20260619 08:22:53.072188 22411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73cdec703c34f569e22678ec33b6523" candidate_uuid: "f867f38697ee4638828375f88dc42eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e297613f294ea9a9e0bf4acf95f1b8"
I20260619 08:22:53.072196 22542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73cdec703c34f569e22678ec33b6523" candidate_uuid: "f867f38697ee4638828375f88dc42eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "109aa9cc9392474fa1733c48c96c1d35"
I20260619 08:22:53.072274 22411 raft_consensus.cc:3037] T c73cdec703c34f569e22678ec33b6523 P 77e297613f294ea9a9e0bf4acf95f1b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:53.072299 22542 raft_consensus.cc:3037] T c73cdec703c34f569e22678ec33b6523 P 109aa9cc9392474fa1733c48c96c1d35 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:53.073060 22411 raft_consensus.cc:2454] T c73cdec703c34f569e22678ec33b6523 P 77e297613f294ea9a9e0bf4acf95f1b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f867f38697ee4638828375f88dc42eaa in term 1.
I20260619 08:22:53.073060 22542 raft_consensus.cc:2454] T c73cdec703c34f569e22678ec33b6523 P 109aa9cc9392474fa1733c48c96c1d35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f867f38697ee4638828375f88dc42eaa in term 1.
I20260619 08:22:53.073243 22607 leader_election.cc:302] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e297613f294ea9a9e0bf4acf95f1b8, f867f38697ee4638828375f88dc42eaa; no voters:
I20260619 08:22:53.073359 22740 raft_consensus.cc:2781] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:22:53.073544 22740 raft_consensus.cc:686] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa [term 1 LEADER]: Becoming Leader. State: Replica: f867f38697ee4638828375f88dc42eaa, State: Running, Role: LEADER
I20260619 08:22:53.073665 22740 consensus_queue.cc:237] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109aa9cc9392474fa1733c48c96c1d35" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37867 } } peers { permanent_uuid: "77e297613f294ea9a9e0bf4acf95f1b8" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 38055 } } peers { permanent_uuid: "f867f38697ee4638828375f88dc42eaa" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 45669 } }
I20260619 08:22:53.074708 22268 catalog_manager.cc:5495] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to f867f38697ee4638828375f88dc42eaa (127.4.151.131). New cstate: current_term: 1 leader_uuid: "f867f38697ee4638828375f88dc42eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109aa9cc9392474fa1733c48c96c1d35" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e297613f294ea9a9e0bf4acf95f1b8" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 38055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f867f38697ee4638828375f88dc42eaa" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 45669 } health_report { overall_health: HEALTHY } } }
I20260619 08:22:53.087826 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:53.085047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43284 (local address 127.4.151.131:45669)
0619 08:22:53.085133 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:53.085135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:53.085172 (+ 37us) server_negotiation.cc:407] Connection header received
0619 08:22:53.085221 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:53.085222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:53.085243 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:53.085332 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:22:53.085532 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:53.086142 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:53.086908 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:53.087091 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:53.087144 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:53.087406 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:22:53.087504 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:22:53.087569 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:53.087677 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 08:22:53.087695 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 08:22:53.090029 22411 raft_consensus.cc:1261] T c73cdec703c34f569e22678ec33b6523 P 77e297613f294ea9a9e0bf4acf95f1b8 [term 1 FOLLOWER]: Refusing update from remote peer f867f38697ee4638828375f88dc42eaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:22:53.090096 22542 raft_consensus.cc:1261] T c73cdec703c34f569e22678ec33b6523 P 109aa9cc9392474fa1733c48c96c1d35 [term 1 FOLLOWER]: Refusing update from remote peer f867f38697ee4638828375f88dc42eaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:22:53.090657 22740 consensus_queue.cc:1035] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e297613f294ea9a9e0bf4acf95f1b8" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 38055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:22:53.090809 22743 consensus_queue.cc:1035] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "109aa9cc9392474fa1733c48c96c1d35" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:22:53.094375 22746 mvcc.cc:205] Tried to move back new op lower bound from 7298487800173346816 to 7298487800110686208. Current Snapshot: MvccSnapshot[applied={T|T < 7298487800173346816 or (T in {7298487800173346816})}]
I20260619 08:22:53.094568 22748 mvcc.cc:205] Tried to move back new op lower bound from 7298487800173346816 to 7298487800110686208. Current Snapshot: MvccSnapshot[applied={T|T < 7298487800173346816 or (T in {7298487800173346816})}]
I20260619 08:22:53.095615 22269 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:22:53.095738 22269 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:22:53.096850 22269 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 27cc149d49db44848225225b33f8ec25: Sending DeleteTablet for 3 replicas of tablet c73cdec703c34f569e22678ec33b6523
I20260619 08:22:53.097235 22522 tablet_service.cc:1514] Processing DeleteTablet for tablet c73cdec703c34f569e22678ec33b6523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:22:53 UTC) from {username='slave'} at 127.0.0.1:38356
I20260619 08:22:53.097288 22653 tablet_service.cc:1514] Processing DeleteTablet for tablet c73cdec703c34f569e22678ec33b6523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:22:53 UTC) from {username='slave'} at 127.0.0.1:43274
I20260619 08:22:53.097536 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 22329
I20260619 08:22:53.097538 22391 tablet_service.cc:1514] Processing DeleteTablet for tablet c73cdec703c34f569e22678ec33b6523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:22:53 UTC) from {username='slave'} at 127.0.0.1:60322
I20260619 08:22:53.098309 22756 tablet_replica.cc:330] T c73cdec703c34f569e22678ec33b6523 P 109aa9cc9392474fa1733c48c96c1d35: stopping tablet replica
I20260619 08:22:53.098438 22756 raft_consensus.cc:2229] T c73cdec703c34f569e22678ec33b6523 P 109aa9cc9392474fa1733c48c96c1d35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:22:53.098548 22756 raft_consensus.cc:2258] T c73cdec703c34f569e22678ec33b6523 P 109aa9cc9392474fa1733c48c96c1d35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:22:53.098738 22755 tablet_replica.cc:330] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa: stopping tablet replica
I20260619 08:22:53.098862 22755 raft_consensus.cc:2229] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:22:53.098991 22755 raft_consensus.cc:2258] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:22:53.099263 22756 ts_tablet_manager.cc:1905] T c73cdec703c34f569e22678ec33b6523 P 109aa9cc9392474fa1733c48c96c1d35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:22:53.101541 22756 ts_tablet_manager.cc:1918] T c73cdec703c34f569e22678ec33b6523 P 109aa9cc9392474fa1733c48c96c1d35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:22:53.101629 22756 log.cc:1196] T c73cdec703c34f569e22678ec33b6523 P 109aa9cc9392474fa1733c48c96c1d35: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/c73cdec703c34f569e22678ec33b6523
I20260619 08:22:53.101895 22756 ts_tablet_manager.cc:1939] T c73cdec703c34f569e22678ec33b6523 P 109aa9cc9392474fa1733c48c96c1d35: Deleting consensus metadata
I20260619 08:22:53.102340 22254 catalog_manager.cc:4841] TS 109aa9cc9392474fa1733c48c96c1d35 (127.4.151.130:37867): tablet c73cdec703c34f569e22678ec33b6523 (table test-table [id=08bbc28b045e4c09bc649caffa67558d]) successfully deleted
I20260619 08:22:53.102671 22755 ts_tablet_manager.cc:1905] T c73cdec703c34f569e22678ec33b6523 P f867f38697ee4638828375f88dc42eaa: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 08:22:53.103227 22254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.151.129:38055 (error 108)
I20260619 08:22:53.103399 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 22460
I20260619 08:22:53.109157 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 22591
I20260619 08:22:53.109252 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:53.109044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:38055 (local address 127.0.0.1:60326)
0619 08:22:53.109162 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:53.109211 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.129:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260619 08:22:53.109406 22254 catalog_manager.cc:4584] TS 77e297613f294ea9a9e0bf4acf95f1b8 (127.4.151.129:38055): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c73cdec703c34f569e22678ec33b6523: Network error: Client connection negotiation failed: client connection to 127.4.151.129:38055: connect: Connection refused (error 111)
I20260619 08:22:53.113926 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 22238
I20260619 08:22:53.114020 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:53.113898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:45669 (local address 127.0.0.1:43296)
0619 08:22:53.113950 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:53.113982 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.131:45669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
2026-06-19T08:22:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (809 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:22:53 dist-test-slave-d3n8 krb5kdc[22760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:22:53 dist-test-slave-d3n8 krb5kdc[22760](info): set up 2 sockets
Jun 19 08:22:53 dist-test-slave-d3n8 krb5kdc[22760](info): commencing operation
krb5kdc: starting...
W20260619 08:22:54.173527 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:22:54 dist-test-slave-d3n8 krb5kdc[22760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:22:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:22:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:22:54.306514 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:45621
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:45621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:22:54.391710 22773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:54.391927 22773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:54.391950 22773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:54.393121 22773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:22:54.393167 22773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:54.393181 22773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:54.393194 22773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:22:54.393206 22773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:22:54.394763 22773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:45621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:45621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:54.394968 22773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:54.395138 22773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:54.400308 22779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:54.400388 22778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:54.400454 22773 server_base.cc:1034] running on GCE node
W20260619 08:22:54.400417 22781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:54.400897 22773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:54.401435 22773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:54.402623 22773 hybrid_clock.cc:648] HybridClock initialized: now 1781857374402591 us; error 45 us; skew 500 ppm
Jun 19 08:22:54 dist-test-slave-d3n8 krb5kdc[22760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857374, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:22:54.405964 22773 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:22:54.406637 22773 webserver.cc:460] Webserver started at http://127.4.151.190:36325/ using document root <none> and password file <none>
I20260619 08:22:54.406917 22773 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:54.406965 22773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:54.407105 22773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:22:54.408279 22773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "7cfefc2e84aa4633a7b653481ff9bb62"
format_stamp: "Formatted at 2026-06-19 08:22:54 on dist-test-slave-d3n8"
server_key: "7e490ca21e859aba45a0f7970f3487cc"
server_key_iv: "0ab7320be612e12450e57c3b792d0a0f"
server_key_version: "encryptionkey@0"
I20260619 08:22:54.408799 22773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "7cfefc2e84aa4633a7b653481ff9bb62"
format_stamp: "Formatted at 2026-06-19 08:22:54 on dist-test-slave-d3n8"
server_key: "7e490ca21e859aba45a0f7970f3487cc"
server_key_iv: "0ab7320be612e12450e57c3b792d0a0f"
server_key_version: "encryptionkey@0"
I20260619 08:22:54.410400 22773 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:22:54.411420 22788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:54.411824 22773 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:22:54.411962 22773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "7cfefc2e84aa4633a7b653481ff9bb62"
format_stamp: "Formatted at 2026-06-19 08:22:54 on dist-test-slave-d3n8"
server_key: "7e490ca21e859aba45a0f7970f3487cc"
server_key_iv: "0ab7320be612e12450e57c3b792d0a0f"
server_key_version: "encryptionkey@0"
I20260619 08:22:54.412061 22773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:54.429100 22773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:54.431802 22773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:54.431919 22773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:54.436079 22773 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:45621
I20260619 08:22:54.436223 22840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:45621 every 8 connection(s)
I20260619 08:22:54.436460 22773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:22:54.437471 22841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:54.439786 22841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cfefc2e84aa4633a7b653481ff9bb62: Bootstrap starting.
I20260619 08:22:54.440342 22841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cfefc2e84aa4633a7b653481ff9bb62: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:54.440575 22841 log.cc:824] T 00000000000000000000000000000000 P 7cfefc2e84aa4633a7b653481ff9bb62: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:54.441476 22841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cfefc2e84aa4633a7b653481ff9bb62: No bootstrap required, opened a new log
I20260619 08:22:54.442003 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 22773
I20260619 08:22:54.442152 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:22:54.442358 4702 external_mini_cluster.cc:1428] Setting key 5463268834afb0906f8addbd251eade6
I20260619 08:22:54.443713 22841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cfefc2e84aa4633a7b653481ff9bb62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfefc2e84aa4633a7b653481ff9bb62" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 45621 } }
I20260619 08:22:54.443903 22841 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cfefc2e84aa4633a7b653481ff9bb62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:54.443943 22841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cfefc2e84aa4633a7b653481ff9bb62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cfefc2e84aa4633a7b653481ff9bb62, State: Initialized, Role: FOLLOWER
I20260619 08:22:54.444103 22841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cfefc2e84aa4633a7b653481ff9bb62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfefc2e84aa4633a7b653481ff9bb62" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 45621 } }
I20260619 08:22:54.444195 22841 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cfefc2e84aa4633a7b653481ff9bb62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:22:54.444227 22841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cfefc2e84aa4633a7b653481ff9bb62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:22:54.444295 22841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cfefc2e84aa4633a7b653481ff9bb62 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:54.445477 22841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cfefc2e84aa4633a7b653481ff9bb62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfefc2e84aa4633a7b653481ff9bb62" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 45621 } }
I20260619 08:22:54.445590 22841 leader_election.cc:302] T 00000000000000000000000000000000 P 7cfefc2e84aa4633a7b653481ff9bb62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cfefc2e84aa4633a7b653481ff9bb62; no voters:
I20260619 08:22:54.445739 22841 leader_election.cc:288] T 00000000000000000000000000000000 P 7cfefc2e84aa4633a7b653481ff9bb62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:22:54.445807 22846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cfefc2e84aa4633a7b653481ff9bb62 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:22:54.445966 22846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cfefc2e84aa4633a7b653481ff9bb62 [term 1 LEADER]: Becoming Leader. State: Replica: 7cfefc2e84aa4633a7b653481ff9bb62, State: Running, Role: LEADER
I20260619 08:22:54.445991 22841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cfefc2e84aa4633a7b653481ff9bb62 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:22:54.446054 22846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cfefc2e84aa4633a7b653481ff9bb62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfefc2e84aa4633a7b653481ff9bb62" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 45621 } }
I20260619 08:22:54.446389 22847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cfefc2e84aa4633a7b653481ff9bb62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cfefc2e84aa4633a7b653481ff9bb62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfefc2e84aa4633a7b653481ff9bb62" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 45621 } } }
I20260619 08:22:54.446472 22847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cfefc2e84aa4633a7b653481ff9bb62 [sys.catalog]: This master's current role is: LEADER
I20260619 08:22:54.446727 22848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cfefc2e84aa4633a7b653481ff9bb62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cfefc2e84aa4633a7b653481ff9bb62. Latest consensus state: current_term: 1 leader_uuid: "7cfefc2e84aa4633a7b653481ff9bb62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfefc2e84aa4633a7b653481ff9bb62" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 45621 } } }
I20260619 08:22:54.446811 22848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cfefc2e84aa4633a7b653481ff9bb62 [sys.catalog]: This master's current role is: LEADER
I20260619 08:22:54.446934 22853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:22:54.447407 22853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:22:54.448742 22863 mvcc.cc:205] Tried to move back new op lower bound from 7298487805740441600 to 7298487805731463168. Current Snapshot: MvccSnapshot[applied={T|T < 7298487805740441600}]
I20260619 08:22:54.449107 22853 catalog_manager.cc:1347] Generated new cluster ID: d84cd1056f9d4c4f856c75ef2bb67d13
I20260619 08:22:54.449155 22853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:22:54 dist-test-slave-d3n8 krb5kdc[22760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:22:54.455267 22845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:54.442926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38214 (local address 127.4.151.190:45621)
0619 08:22:54.443187 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:54.443198 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:54.443233 (+ 35us) server_negotiation.cc:407] Connection header received
0619 08:22:54.443667 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:54.443684 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:54.443750 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:54.443907 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:54.444809 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:54.446370 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:54.447523 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:54.448064 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:54.448217 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:54.451183 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:54.451186 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:54.451187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:22:54.451198 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:54.453332 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:54.453775 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:54.453778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:54.453778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:54.453923 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:54.454139 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:54.454140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:54.454140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:54.454459 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:54.454630 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:54.454700 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 08:22:54.454860 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":89,"threads_started":1}
I20260619 08:22:54.456012 22853 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:22:54.456554 22853 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:22:54.460516 22853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cfefc2e84aa4633a7b653481ff9bb62: Generated new TSK 0
I20260619 08:22:54.460703 22853 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.151.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
I20260619 08:22:54.493125 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:45621
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:22:54.578555 22867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:54.578836 22867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:54.578879 22867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:54.580261 22867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:54.580335 22867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:54.580382 22867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:22:54.581976 22867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:45621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:54.582242 22867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:54.582476 22867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:54.587767 22875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:54.587767 22872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:54.587872 22867 server_base.cc:1034] running on GCE node
W20260619 08:22:54.587805 22873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:54.588356 22867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:54.588572 22867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:54.589778 22867 hybrid_clock.cc:648] HybridClock initialized: now 1781857374589754 us; error 49 us; skew 500 ppm
Jun 19 08:22:54 dist-test-slave-d3n8 krb5kdc[22760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857374, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:22:54.592541 22867 init.cc:377] Logged in from keytab as kudu/127.4.151.129@KRBTEST.COM (short username kudu)
I20260619 08:22:54.592999 22867 webserver.cc:460] Webserver started at http://127.4.151.129:37529/ using document root <none> and password file <none>
I20260619 08:22:54.593254 22867 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:54.593323 22867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:54.593483 22867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:22:54.594570 22867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "f34059f2aa7c47e4ad7283097b0628ce"
format_stamp: "Formatted at 2026-06-19 08:22:54 on dist-test-slave-d3n8"
server_key: "639aebe37b658d6e8451893f687a1ebb"
server_key_iv: "1053e3dd32b0cb7d16bbd47e4f49b3ec"
server_key_version: "encryptionkey@0"
I20260619 08:22:54.594946 22867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "f34059f2aa7c47e4ad7283097b0628ce"
format_stamp: "Formatted at 2026-06-19 08:22:54 on dist-test-slave-d3n8"
server_key: "639aebe37b658d6e8451893f687a1ebb"
server_key_iv: "1053e3dd32b0cb7d16bbd47e4f49b3ec"
server_key_version: "encryptionkey@0"
I20260619 08:22:54.596458 22867 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:22:54.597249 22882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:54.597500 22867 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:22:54.597602 22867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "f34059f2aa7c47e4ad7283097b0628ce"
format_stamp: "Formatted at 2026-06-19 08:22:54 on dist-test-slave-d3n8"
server_key: "639aebe37b658d6e8451893f687a1ebb"
server_key_iv: "1053e3dd32b0cb7d16bbd47e4f49b3ec"
server_key_version: "encryptionkey@0"
I20260619 08:22:54.597687 22867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:54.617241 22867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:54.619645 22867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:54.619820 22867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:54.620069 22867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:22:54.620410 22867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:22:54.620469 22867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:54.620513 22867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:22:54.620546 22867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:54.627082 22867 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:42749
I20260619 08:22:54.627159 22995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:42749 every 8 connection(s)
I20260619 08:22:54.627512 22867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
I20260619 08:22:54.629029 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 22867
I20260619 08:22:54.629124 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:22:54.629458 4702 external_mini_cluster.cc:1428] Setting key 49b0c1c9514fa744ae7ba31542503491
Jun 19 08:22:54 dist-test-slave-d3n8 krb5kdc[22760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857374, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.151.130@KRBTEST.COM; defaulting to no policy
I20260619 08:22:54.638993 22845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:54.628424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:45161 (local address 127.4.151.190:45621)
0619 08:22:54.628517 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:54.628519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:54.628719 (+ 200us) server_negotiation.cc:407] Connection header received
0619 08:22:54.629286 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:54.629287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:54.629305 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:54.629362 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:54.630546 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:54.631081 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:54.632636 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:54.632979 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:54.633056 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:54.635564 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:54.635566 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:54.635566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:22:54.635575 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:54.637362 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:54.637823 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:54.637824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:54.637825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:54.637906 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:54.638224 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:54.638225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:54.638225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:54.638369 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:54.638441 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:54.638828 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 08:22:54.638922 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 08:22:54.639137 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:54.628396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:45621 (local address 127.4.151.129:45161)
0619 08:22:54.628574 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:54.628617 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:54.629130 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:54.629446 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:54.629450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:54.630218 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:22:54.630439 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:54.630444 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:54.631199 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:54.631200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:54.632525 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:54.632528 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:54.633130 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:54.633131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:54.633272 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:54.633956 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:22:54.633987 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:22:54.635463 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:22:54.637468 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:54.637476 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:54.637478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:54.637740 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:54.638003 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:54.638004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:54.638004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:54.638132 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:54.638459 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:22:54.638466 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:22:54.638631 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 08:22:54.638784 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 08:22:54.638932 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260619 08:22:54.639895 22996 heartbeater.cc:347] Connected to a master server at 127.4.151.190:45621
I20260619 08:22:54.640017 22996 heartbeater.cc:464] Registering TS with master...
I20260619 08:22:54.640239 22996 heartbeater.cc:511] Master 127.4.151.190:45621 requested a full tablet report, sending...
I20260619 08:22:54.640795 22805 ts_manager.cc:194] Registered new tserver with Master: f34059f2aa7c47e4ad7283097b0628ce (127.4.151.129:42749)
I20260619 08:22:54.641580 22805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.129@KRBTEST.COM'} at 127.4.151.129:45161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
I20260619 08:22:54.661186 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:45621
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:22:54.744129 23001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:54.744350 23001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:54.744374 23001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:54.745502 23001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:54.745579 23001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:54.745626 23001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:22:54.747650 23001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:45621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:54.747969 23001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:54.748303 23001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:54.754106 23007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:54.754119 23006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:54.754120 23009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:54.754256 23001 server_base.cc:1034] running on GCE node
I20260619 08:22:54.754738 23001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:54.755043 23001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:54.756223 23001 hybrid_clock.cc:648] HybridClock initialized: now 1781857374756223 us; error 47 us; skew 500 ppm
Jun 19 08:22:54 dist-test-slave-d3n8 krb5kdc[22760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857374, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:22:54.758957 23001 init.cc:377] Logged in from keytab as kudu/127.4.151.130@KRBTEST.COM (short username kudu)
I20260619 08:22:54.759483 23001 webserver.cc:460] Webserver started at http://127.4.151.130:34811/ using document root <none> and password file <none>
I20260619 08:22:54.759778 23001 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:54.759859 23001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:54.760017 23001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:22:54.761107 23001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "0531976adc1c464aa31b1e9125aa6e92"
format_stamp: "Formatted at 2026-06-19 08:22:54 on dist-test-slave-d3n8"
server_key: "5ef715abd7de241ed4ec7568b427adac"
server_key_iv: "e8e2daf4c6fb4a4d71dcf5236a6123f9"
server_key_version: "encryptionkey@0"
I20260619 08:22:54.761550 23001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "0531976adc1c464aa31b1e9125aa6e92"
format_stamp: "Formatted at 2026-06-19 08:22:54 on dist-test-slave-d3n8"
server_key: "5ef715abd7de241ed4ec7568b427adac"
server_key_iv: "e8e2daf4c6fb4a4d71dcf5236a6123f9"
server_key_version: "encryptionkey@0"
I20260619 08:22:54.762998 23001 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:22:54.763928 23016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:54.764228 23001 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:22:54.764343 23001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "0531976adc1c464aa31b1e9125aa6e92"
format_stamp: "Formatted at 2026-06-19 08:22:54 on dist-test-slave-d3n8"
server_key: "5ef715abd7de241ed4ec7568b427adac"
server_key_iv: "e8e2daf4c6fb4a4d71dcf5236a6123f9"
server_key_version: "encryptionkey@0"
I20260619 08:22:54.764446 23001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:54.809144 23001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:54.811896 23001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:54.812084 23001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:54.812428 23001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:22:54.812772 23001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:22:54.812830 23001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:54.812873 23001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:22:54.812897 23001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:54.819563 23001 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:39659
I20260619 08:22:54.819793 23129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:39659 every 8 connection(s)
I20260619 08:22:54.819976 23001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
Jun 19 08:22:54 dist-test-slave-d3n8 krb5kdc[22760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857374, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:22:54.827338 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 23001
I20260619 08:22:54.827464 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:22:54.827675 4702 external_mini_cluster.cc:1428] Setting key 74dd3f81fdf40e34fec65f429e0d8786
I20260619 08:22:54.830571 22845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:54.820782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:36425 (local address 127.4.151.190:45621)
0619 08:22:54.820864 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:54.820866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:54.821061 (+ 195us) server_negotiation.cc:407] Connection header received
0619 08:22:54.821700 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:54.821752 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:54.821768 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:54.821861 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:54.822992 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:54.823631 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:54.824766 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:54.824983 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:54.825086 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:54.827550 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:54.827554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:54.827554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:22:54.827566 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:54.829019 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:54.829506 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:54.829508 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:54.829508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:54.829568 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:54.829854 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:54.829855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:54.829856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:54.829977 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:54.830057 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:54.830378 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 08:22:54.830469 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 08:22:54.830897 23132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:54.820789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:45621 (local address 127.4.151.130:36425)
0619 08:22:54.820950 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:54.820984 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:54.821573 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:54.821989 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:54.821993 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:54.822654 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:22:54.822867 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:54.822872 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:54.823743 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:54.823745 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:54.824650 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:54.824652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:54.825089 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:54.825090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:54.825234 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:54.825830 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:22:54.825839 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:22:54.827418 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:22:54.829137 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:54.829144 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:54.829147 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:54.829432 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:54.829653 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:54.829664 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:54.829664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:54.829778 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:54.830065 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:22:54.830077 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:22:54.830235 (+ 158us) client_negotiation.cc:769] Sending connection context
0619 08:22:54.830342 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 08:22:54.830514 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260619 08:22:54.831483 23130 heartbeater.cc:347] Connected to a master server at 127.4.151.190:45621
I20260619 08:22:54.831593 23130 heartbeater.cc:464] Registering TS with master...
I20260619 08:22:54.831797 23130 heartbeater.cc:511] Master 127.4.151.190:45621 requested a full tablet report, sending...
I20260619 08:22:54.832288 22805 ts_manager.cc:194] Registered new tserver with Master: 0531976adc1c464aa31b1e9125aa6e92 (127.4.151.130:39659)
I20260619 08:22:54.832762 22805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.130@KRBTEST.COM'} at 127.4.151.130:36425
WARNING: no policy specified for kudu/127.4.151.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
I20260619 08:22:54.857236 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:45621
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:22:54.939934 23135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:54.940155 23135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:54.940179 23135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:54.941347 23135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:54.941407 23135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:54.941435 23135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:22:54.942962 23135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:45621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:54.943162 23135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:54.943344 23135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:54.948625 23141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:54.948637 23143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:54.948661 23140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:54.948889 23135 server_base.cc:1034] running on GCE node
I20260619 08:22:54.949131 23135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:54.949492 23135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:54.950680 23135 hybrid_clock.cc:648] HybridClock initialized: now 1781857374950661 us; error 49 us; skew 500 ppm
Jun 19 08:22:54 dist-test-slave-d3n8 krb5kdc[22760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857374, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:22:54.953364 23135 init.cc:377] Logged in from keytab as kudu/127.4.151.131@KRBTEST.COM (short username kudu)
I20260619 08:22:54.953967 23135 webserver.cc:460] Webserver started at http://127.4.151.131:34445/ using document root <none> and password file <none>
I20260619 08:22:54.954254 23135 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:54.954345 23135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:54.954514 23135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:22:54.955884 23135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "2d2b3c0711a8470c93333621fe28a176"
format_stamp: "Formatted at 2026-06-19 08:22:54 on dist-test-slave-d3n8"
server_key: "220dc7aa982172759029b767914490d1"
server_key_iv: "ca59be46efd2e7ed23ba14d990a490e3"
server_key_version: "encryptionkey@0"
I20260619 08:22:54.956278 23135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "2d2b3c0711a8470c93333621fe28a176"
format_stamp: "Formatted at 2026-06-19 08:22:54 on dist-test-slave-d3n8"
server_key: "220dc7aa982172759029b767914490d1"
server_key_iv: "ca59be46efd2e7ed23ba14d990a490e3"
server_key_version: "encryptionkey@0"
I20260619 08:22:54.957945 23135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260619 08:22:54.958864 23150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:54.959120 23135 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:22:54.959270 23135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "2d2b3c0711a8470c93333621fe28a176"
format_stamp: "Formatted at 2026-06-19 08:22:54 on dist-test-slave-d3n8"
server_key: "220dc7aa982172759029b767914490d1"
server_key_iv: "ca59be46efd2e7ed23ba14d990a490e3"
server_key_version: "encryptionkey@0"
I20260619 08:22:54.959367 23135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:54.991493 23135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:54.994290 23135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:54.994511 23135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:54.994796 23135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:22:54.995150 23135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:22:54.995213 23135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:54.995262 23135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:22:54.995299 23135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:55.002573 23135 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:43651
I20260619 08:22:55.002614 23263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:43651 every 8 connection(s)
I20260619 08:22:55.003034 23135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
Jun 19 08:22:55 dist-test-slave-d3n8 krb5kdc[22760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857374, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:22:55.013024 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 23135
I20260619 08:22:55.013137 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:22:55.013345 4702 external_mini_cluster.cc:1428] Setting key 0827ed80b20b585fba039d4dbb6ebafb
I20260619 08:22:55.014200 22845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.003937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:34493 (local address 127.4.151.190:45621)
0619 08:22:55.004013 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:55.004016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:55.004242 (+ 226us) server_negotiation.cc:407] Connection header received
0619 08:22:55.004709 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:55.004710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:55.004724 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:55.004810 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:55.005901 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.006491 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.007646 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.007860 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.007929 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.010579 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:55.010580 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:55.010581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:22:55.010589 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:55.012334 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:55.012843 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:55.012845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:55.012845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:55.012915 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:55.013205 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:55.013206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:55.013207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:55.013432 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:55.013509 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:55.013952 (+ 443us) server_negotiation.cc:299] Negotiation successful
0619 08:22:55.014092 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 08:22:55.014608 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.003910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:45621 (local address 127.4.151.131:34493)
0619 08:22:55.004075 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:55.004127 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:55.004611 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:55.004867 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:55.004870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:55.005544 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:22:55.005755 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:55.005760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.006595 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.006596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:55.007531 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:55.007535 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.007963 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.007964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:55.008086 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.008725 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:22:55.008733 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:22:55.010460 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:22:55.012455 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:55.012464 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:55.012467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:55.012750 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:55.012995 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:55.012996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:55.012997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:55.013126 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:55.013515 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:22:55.013522 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:22:55.013708 (+ 186us) client_negotiation.cc:769] Sending connection context
0619 08:22:55.013835 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 08:22:55.014030 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260619 08:22:55.015053 23264 heartbeater.cc:347] Connected to a master server at 127.4.151.190:45621
I20260619 08:22:55.015159 23264 heartbeater.cc:464] Registering TS with master...
I20260619 08:22:55.015316 23264 heartbeater.cc:511] Master 127.4.151.190:45621 requested a full tablet report, sending...
I20260619 08:22:55.015849 22805 ts_manager.cc:194] Registered new tserver with Master: 2d2b3c0711a8470c93333621fe28a176 (127.4.151.131:43651)
I20260619 08:22:55.016194 22805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.131@KRBTEST.COM'} at 127.4.151.131:34493
I20260619 08:22:55.025776 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:22:55.034448 22845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.027204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.4.151.190:45621)
0619 08:22:55.027274 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:55.027277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:55.027454 (+ 177us) server_negotiation.cc:407] Connection header received
0619 08:22:55.027540 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:55.027541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:55.027552 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:55.027633 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:55.028262 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.028813 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.029691 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.029910 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.030004 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.031546 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:55.031547 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:55.031547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:22:55.031555 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:55.033019 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:55.033516 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:55.033518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:55.033518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:55.033583 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:55.033832 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:55.033833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:55.033834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:55.033985 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:55.034118 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:55.034266 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 08:22:55.034339 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 08:22:55.036324 22805 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38230:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:22:55.038074 22805 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:22:55.044566 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.041108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47916 (local address 127.4.151.129:42749)
0619 08:22:55.041383 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:55.041388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:55.041401 (+ 13us) server_negotiation.cc:407] Connection header received
0619 08:22:55.041444 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:55.041446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:55.041462 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:55.041535 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:55.041635 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.042435 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.043531 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.044130 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.044207 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.044277 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:55.044434 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 08:22:55.044475 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":102,"threads_started":1}
I20260619 08:22:55.044570 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.040628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:42749 (local address 127.0.0.1:47916)
0619 08:22:55.040771 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:55.040816 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:55.040885 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:55.041530 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:55.041531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:55.041533 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:55.041579 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:55.041581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.042542 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.042543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:55.043430 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:55.043432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.044218 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.044219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:55.044312 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.044327 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 08:22:55.044402 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 08:22:55.044467 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":64,"threads_started":1}
I20260619 08:22:55.045547 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.040631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:39659 (local address 127.0.0.1:38976)
0619 08:22:55.040787 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:55.040816 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:55.040885 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:55.041229 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:55.041232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:55.041241 (+ 9us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:55.041395 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:55.041399 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.042758 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.042759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:55.043881 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:55.043884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.045189 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.045190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:55.045311 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.045317 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 08:22:55.045422 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 08:22:55.045469 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
I20260619 08:22:55.045562 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.040912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38976 (local address 127.4.151.130:39659)
0619 08:22:55.041066 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:55.041071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:55.041078 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:22:55.041109 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:55.041111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:55.041123 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:55.041208 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:55.041487 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.042630 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.043976 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.045091 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.045185 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.045240 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:55.045442 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 08:22:55.045484 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260619 08:22:55.045459 22930 tablet_service.cc:1467] Processing CreateTablet for tablet 9e2933d8d3f649e694564faa9808ea2f (DEFAULT_TABLE table=test-table [id=290077bac21c48d98310fd2f11395072]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:22:55.045806 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.040903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54584 (local address 127.4.151.131:43651)
0619 08:22:55.041669 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:55.041673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:55.041684 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:22:55.041723 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:55.041725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:55.041750 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:55.041831 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:55.041951 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.042990 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.044478 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.045182 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.045251 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.045327 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:55.045670 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 08:22:55.045713 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":91,"threads_started":1}
I20260619 08:22:55.045871 22930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e2933d8d3f649e694564faa9808ea2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:55.046041 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.040636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:43651 (local address 127.0.0.1:54584)
0619 08:22:55.041386 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:55.041397 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:55.041452 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:55.041847 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:55.041848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:55.041851 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:55.041888 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:55.041889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.043099 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.043099 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:55.044392 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:55.044395 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.045365 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.045366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:55.045454 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.045458 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:22:55.045511 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 08:22:55.045944 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":87,"threads_started":1}
I20260619 08:22:55.046722 23198 tablet_service.cc:1467] Processing CreateTablet for tablet 9e2933d8d3f649e694564faa9808ea2f (DEFAULT_TABLE table=test-table [id=290077bac21c48d98310fd2f11395072]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:22:55.047005 23198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e2933d8d3f649e694564faa9808ea2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:55.048813 23281 tablet_bootstrap.cc:492] T 9e2933d8d3f649e694564faa9808ea2f P f34059f2aa7c47e4ad7283097b0628ce: Bootstrap starting.
I20260619 08:22:55.049392 23064 tablet_service.cc:1467] Processing CreateTablet for tablet 9e2933d8d3f649e694564faa9808ea2f (DEFAULT_TABLE table=test-table [id=290077bac21c48d98310fd2f11395072]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:22:55.049510 23281 tablet_bootstrap.cc:654] T 9e2933d8d3f649e694564faa9808ea2f P f34059f2aa7c47e4ad7283097b0628ce: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:55.049727 23064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e2933d8d3f649e694564faa9808ea2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:55.050097 23281 log.cc:824] T 9e2933d8d3f649e694564faa9808ea2f P f34059f2aa7c47e4ad7283097b0628ce: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:55.050299 23282 tablet_bootstrap.cc:492] T 9e2933d8d3f649e694564faa9808ea2f P 2d2b3c0711a8470c93333621fe28a176: Bootstrap starting.
I20260619 08:22:55.050792 23281 tablet_bootstrap.cc:492] T 9e2933d8d3f649e694564faa9808ea2f P f34059f2aa7c47e4ad7283097b0628ce: No bootstrap required, opened a new log
I20260619 08:22:55.050848 23281 ts_tablet_manager.cc:1397] T 9e2933d8d3f649e694564faa9808ea2f P f34059f2aa7c47e4ad7283097b0628ce: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 08:22:55.051060 23282 tablet_bootstrap.cc:654] T 9e2933d8d3f649e694564faa9808ea2f P 2d2b3c0711a8470c93333621fe28a176: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:55.051628 23282 log.cc:824] T 9e2933d8d3f649e694564faa9808ea2f P 2d2b3c0711a8470c93333621fe28a176: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:55.052561 23282 tablet_bootstrap.cc:492] T 9e2933d8d3f649e694564faa9808ea2f P 2d2b3c0711a8470c93333621fe28a176: No bootstrap required, opened a new log
I20260619 08:22:55.052636 23282 ts_tablet_manager.cc:1397] T 9e2933d8d3f649e694564faa9808ea2f P 2d2b3c0711a8470c93333621fe28a176: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:22:55.052520 23281 raft_consensus.cc:348] T 9e2933d8d3f649e694564faa9808ea2f P f34059f2aa7c47e4ad7283097b0628ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34059f2aa7c47e4ad7283097b0628ce" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42749 } } peers { permanent_uuid: "0531976adc1c464aa31b1e9125aa6e92" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39659 } } peers { permanent_uuid: "2d2b3c0711a8470c93333621fe28a176" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 43651 } }
I20260619 08:22:55.052698 23281 raft_consensus.cc:374] T 9e2933d8d3f649e694564faa9808ea2f P f34059f2aa7c47e4ad7283097b0628ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:55.052742 23281 raft_consensus.cc:729] T 9e2933d8d3f649e694564faa9808ea2f P f34059f2aa7c47e4ad7283097b0628ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f34059f2aa7c47e4ad7283097b0628ce, State: Initialized, Role: FOLLOWER
I20260619 08:22:55.052855 23281 consensus_queue.cc:260] T 9e2933d8d3f649e694564faa9808ea2f P f34059f2aa7c47e4ad7283097b0628ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34059f2aa7c47e4ad7283097b0628ce" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42749 } } peers { permanent_uuid: "0531976adc1c464aa31b1e9125aa6e92" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39659 } } peers { permanent_uuid: "2d2b3c0711a8470c93333621fe28a176" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 43651 } }
I20260619 08:22:55.053167 23281 ts_tablet_manager.cc:1428] T 9e2933d8d3f649e694564faa9808ea2f P f34059f2aa7c47e4ad7283097b0628ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 08:22:55.053288 23286 tablet_bootstrap.cc:492] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92: Bootstrap starting.
I20260619 08:22:55.054301 23286 tablet_bootstrap.cc:654] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:55.054437 23282 raft_consensus.cc:348] T 9e2933d8d3f649e694564faa9808ea2f P 2d2b3c0711a8470c93333621fe28a176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34059f2aa7c47e4ad7283097b0628ce" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42749 } } peers { permanent_uuid: "0531976adc1c464aa31b1e9125aa6e92" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39659 } } peers { permanent_uuid: "2d2b3c0711a8470c93333621fe28a176" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 43651 } }
I20260619 08:22:55.054587 23282 raft_consensus.cc:374] T 9e2933d8d3f649e694564faa9808ea2f P 2d2b3c0711a8470c93333621fe28a176 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:55.054610 23282 raft_consensus.cc:729] T 9e2933d8d3f649e694564faa9808ea2f P 2d2b3c0711a8470c93333621fe28a176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d2b3c0711a8470c93333621fe28a176, State: Initialized, Role: FOLLOWER
I20260619 08:22:55.054733 23282 consensus_queue.cc:260] T 9e2933d8d3f649e694564faa9808ea2f P 2d2b3c0711a8470c93333621fe28a176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34059f2aa7c47e4ad7283097b0628ce" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42749 } } peers { permanent_uuid: "0531976adc1c464aa31b1e9125aa6e92" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39659 } } peers { permanent_uuid: "2d2b3c0711a8470c93333621fe28a176" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 43651 } }
I20260619 08:22:55.055015 23282 ts_tablet_manager.cc:1428] T 9e2933d8d3f649e694564faa9808ea2f P 2d2b3c0711a8470c93333621fe28a176: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 08:22:55.055022 23286 log.cc:824] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:55.055051 23264 heartbeater.cc:503] Master 127.4.151.190:45621 was elected leader, sending a full tablet report...
I20260619 08:22:55.055316 22996 heartbeater.cc:503] Master 127.4.151.190:45621 was elected leader, sending a full tablet report...
I20260619 08:22:55.056160 23286 tablet_bootstrap.cc:492] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92: No bootstrap required, opened a new log
I20260619 08:22:55.056233 23286 ts_tablet_manager.cc:1397] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 08:22:55.057704 23286 raft_consensus.cc:348] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34059f2aa7c47e4ad7283097b0628ce" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42749 } } peers { permanent_uuid: "0531976adc1c464aa31b1e9125aa6e92" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39659 } } peers { permanent_uuid: "2d2b3c0711a8470c93333621fe28a176" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 43651 } }
I20260619 08:22:55.057875 23286 raft_consensus.cc:374] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:55.057922 23286 raft_consensus.cc:729] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0531976adc1c464aa31b1e9125aa6e92, State: Initialized, Role: FOLLOWER
I20260619 08:22:55.058045 23286 consensus_queue.cc:260] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34059f2aa7c47e4ad7283097b0628ce" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42749 } } peers { permanent_uuid: "0531976adc1c464aa31b1e9125aa6e92" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39659 } } peers { permanent_uuid: "2d2b3c0711a8470c93333621fe28a176" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 43651 } }
I20260619 08:22:55.058296 23286 ts_tablet_manager.cc:1428] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 08:22:55.058350 23130 heartbeater.cc:503] Master 127.4.151.190:45621 was elected leader, sending a full tablet report...
W20260619 08:22:55.070704 23131 tablet.cc:2369] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:22:55.128271 22997 tablet.cc:2369] T 9e2933d8d3f649e694564faa9808ea2f P f34059f2aa7c47e4ad7283097b0628ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:22:55.253661 23265 tablet.cc:2369] T 9e2933d8d3f649e694564faa9808ea2f P 2d2b3c0711a8470c93333621fe28a176: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:22:55.274164 23289 raft_consensus.cc:482] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:22:55.274299 23289 raft_consensus.cc:504] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34059f2aa7c47e4ad7283097b0628ce" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42749 } } peers { permanent_uuid: "0531976adc1c464aa31b1e9125aa6e92" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39659 } } peers { permanent_uuid: "2d2b3c0711a8470c93333621fe28a176" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 43651 } }
I20260619 08:22:55.274622 23289 leader_election.cc:288] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f34059f2aa7c47e4ad7283097b0628ce (127.4.151.129:42749), 2d2b3c0711a8470c93333621fe28a176 (127.4.151.131:43651)
I20260619 08:22:55.277825 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.274894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:39379 (local address 127.4.151.131:43651)
0619 08:22:55.274940 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:55.274942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:55.275041 (+ 99us) server_negotiation.cc:407] Connection header received
0619 08:22:55.275110 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:55.275110 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:55.275138 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:55.275196 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:55.275465 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.275924 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.277019 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.277461 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.277547 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.277558 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:55.277704 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 08:22:55.277753 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 08:22:55.277832 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.274903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:50237 (local address 127.4.151.129:42749)
0619 08:22:55.274954 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:55.274956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:55.274968 (+ 12us) server_negotiation.cc:407] Connection header received
0619 08:22:55.275076 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:55.275077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:55.275088 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:55.275144 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:55.275393 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.275935 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.276978 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.277462 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.277547 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.277556 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:55.277733 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 08:22:55.277771 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 08:22:55.277832 23132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.274822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:42749 (local address 127.4.151.130:50237)
0619 08:22:55.274898 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:55.274912 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:55.274971 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:55.275163 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:55.275164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:55.275165 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:55.275306 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:55.275309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.276054 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.276055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:55.276865 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:55.276867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.277546 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.277547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:55.277640 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.277643 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:22:55.277719 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 08:22:55.277759 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":3}
I20260619 08:22:55.277837 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.274822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:43651 (local address 127.4.151.130:39379)
0619 08:22:55.274958 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:55.274968 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:55.275034 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:55.275285 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:55.275285 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:55.275287 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:55.275373 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:55.275375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.276018 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.276019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:55.276896 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:55.276899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.277546 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.277547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:55.277627 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.277630 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:22:55.277676 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 08:22:55.277740 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":106,"thread_start_us":62,"threads_started":1}
I20260619 08:22:55.278174 22950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e2933d8d3f649e694564faa9808ea2f" candidate_uuid: "0531976adc1c464aa31b1e9125aa6e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f34059f2aa7c47e4ad7283097b0628ce" is_pre_election: true
I20260619 08:22:55.278350 22950 raft_consensus.cc:2454] T 9e2933d8d3f649e694564faa9808ea2f P f34059f2aa7c47e4ad7283097b0628ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0531976adc1c464aa31b1e9125aa6e92 in term 0.
I20260619 08:22:55.278385 23218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e2933d8d3f649e694564faa9808ea2f" candidate_uuid: "0531976adc1c464aa31b1e9125aa6e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d2b3c0711a8470c93333621fe28a176" is_pre_election: true
I20260619 08:22:55.278525 23218 raft_consensus.cc:2454] T 9e2933d8d3f649e694564faa9808ea2f P 2d2b3c0711a8470c93333621fe28a176 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0531976adc1c464aa31b1e9125aa6e92 in term 0.
I20260619 08:22:55.278759 23018 leader_election.cc:302] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0531976adc1c464aa31b1e9125aa6e92, 2d2b3c0711a8470c93333621fe28a176; no voters:
I20260619 08:22:55.278945 23289 raft_consensus.cc:2781] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:22:55.279151 23289 raft_consensus.cc:482] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:22:55.279218 23289 raft_consensus.cc:3037] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:55.279965 23289 raft_consensus.cc:504] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34059f2aa7c47e4ad7283097b0628ce" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42749 } } peers { permanent_uuid: "0531976adc1c464aa31b1e9125aa6e92" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39659 } } peers { permanent_uuid: "2d2b3c0711a8470c93333621fe28a176" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 43651 } }
I20260619 08:22:55.280139 23289 leader_election.cc:288] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92 [CANDIDATE]: Term 1 election: Requested vote from peers f34059f2aa7c47e4ad7283097b0628ce (127.4.151.129:42749), 2d2b3c0711a8470c93333621fe28a176 (127.4.151.131:43651)
I20260619 08:22:55.280283 23218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e2933d8d3f649e694564faa9808ea2f" candidate_uuid: "0531976adc1c464aa31b1e9125aa6e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d2b3c0711a8470c93333621fe28a176"
I20260619 08:22:55.280369 22950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e2933d8d3f649e694564faa9808ea2f" candidate_uuid: "0531976adc1c464aa31b1e9125aa6e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f34059f2aa7c47e4ad7283097b0628ce"
I20260619 08:22:55.280373 23218 raft_consensus.cc:3037] T 9e2933d8d3f649e694564faa9808ea2f P 2d2b3c0711a8470c93333621fe28a176 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:55.280439 22950 raft_consensus.cc:3037] T 9e2933d8d3f649e694564faa9808ea2f P f34059f2aa7c47e4ad7283097b0628ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:55.281261 22950 raft_consensus.cc:2454] T 9e2933d8d3f649e694564faa9808ea2f P f34059f2aa7c47e4ad7283097b0628ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0531976adc1c464aa31b1e9125aa6e92 in term 1.
I20260619 08:22:55.281260 23218 raft_consensus.cc:2454] T 9e2933d8d3f649e694564faa9808ea2f P 2d2b3c0711a8470c93333621fe28a176 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0531976adc1c464aa31b1e9125aa6e92 in term 1.
I20260619 08:22:55.281461 23018 leader_election.cc:302] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0531976adc1c464aa31b1e9125aa6e92, 2d2b3c0711a8470c93333621fe28a176; no voters:
I20260619 08:22:55.281579 23289 raft_consensus.cc:2781] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:22:55.281734 23289 raft_consensus.cc:686] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92 [term 1 LEADER]: Becoming Leader. State: Replica: 0531976adc1c464aa31b1e9125aa6e92, State: Running, Role: LEADER
I20260619 08:22:55.281845 23289 consensus_queue.cc:237] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34059f2aa7c47e4ad7283097b0628ce" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42749 } } peers { permanent_uuid: "0531976adc1c464aa31b1e9125aa6e92" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39659 } } peers { permanent_uuid: "2d2b3c0711a8470c93333621fe28a176" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 43651 } }
I20260619 08:22:55.282724 22804 catalog_manager.cc:5495] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0531976adc1c464aa31b1e9125aa6e92 (127.4.151.130). New cstate: current_term: 1 leader_uuid: "0531976adc1c464aa31b1e9125aa6e92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34059f2aa7c47e4ad7283097b0628ce" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0531976adc1c464aa31b1e9125aa6e92" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d2b3c0711a8470c93333621fe28a176" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 43651 } health_report { overall_health: UNKNOWN } } }
I20260619 08:22:55.332352 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.329774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38980 (local address 127.4.151.130:39659)
0619 08:22:55.329859 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:55.329861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:55.329877 (+ 16us) server_negotiation.cc:407] Connection header received
0619 08:22:55.329913 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:55.329913 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:55.329931 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:55.329995 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:22:55.330188 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.330669 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.331433 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.331611 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.331661 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.331851 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:22:55.331937 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:22:55.332048 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:55.332176 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 08:22:55.332210 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 08:22:55.334218 23218 raft_consensus.cc:1261] T 9e2933d8d3f649e694564faa9808ea2f P 2d2b3c0711a8470c93333621fe28a176 [term 1 FOLLOWER]: Refusing update from remote peer 0531976adc1c464aa31b1e9125aa6e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:22:55.334218 22950 raft_consensus.cc:1261] T 9e2933d8d3f649e694564faa9808ea2f P f34059f2aa7c47e4ad7283097b0628ce [term 1 FOLLOWER]: Refusing update from remote peer 0531976adc1c464aa31b1e9125aa6e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:22:55.334645 23291 consensus_queue.cc:1035] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f34059f2aa7c47e4ad7283097b0628ce" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 42749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:22:55.334789 23289 consensus_queue.cc:1035] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d2b3c0711a8470c93333621fe28a176" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 43651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:22:55.337909 23296 mvcc.cc:205] Tried to move back new op lower bound from 7298487809365991424 to 7298487809155047424. Current Snapshot: MvccSnapshot[applied={T|T < 7298487809365991424 or (T in {7298487809365991424})}]
I20260619 08:22:55.338084 23295 mvcc.cc:205] Tried to move back new op lower bound from 7298487809365991424 to 7298487809155047424. Current Snapshot: MvccSnapshot[applied={T|T < 7298487809365991424}]
I20260619 08:22:55.339309 22804 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:22:55.339417 22804 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:22:55.340167 22804 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7cfefc2e84aa4633a7b653481ff9bb62: Sending DeleteTablet for 3 replicas of tablet 9e2933d8d3f649e694564faa9808ea2f
I20260619 08:22:55.340502 22930 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e2933d8d3f649e694564faa9808ea2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:22:55 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:47916
I20260619 08:22:55.340518 23198 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e2933d8d3f649e694564faa9808ea2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:22:55 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:54584
I20260619 08:22:55.340936 23304 tablet_replica.cc:330] T 9e2933d8d3f649e694564faa9808ea2f P f34059f2aa7c47e4ad7283097b0628ce: stopping tablet replica
I20260619 08:22:55.341048 23304 raft_consensus.cc:2229] T 9e2933d8d3f649e694564faa9808ea2f P f34059f2aa7c47e4ad7283097b0628ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:22:55.341030 23064 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e2933d8d3f649e694564faa9808ea2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:22:55 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:38976
I20260619 08:22:55.341166 23304 raft_consensus.cc:2258] T 9e2933d8d3f649e694564faa9808ea2f P f34059f2aa7c47e4ad7283097b0628ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:22:55.341262 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 22867
I20260619 08:22:55.342115 23306 tablet_replica.cc:330] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92: stopping tablet replica
I20260619 08:22:55.342250 23306 raft_consensus.cc:2229] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:22:55.342412 23306 raft_consensus.cc:2258] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:22:55.342877 23306 ts_tablet_manager.cc:1905] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:22:55.344425 23306 ts_tablet_manager.cc:1918] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:22:55.344513 23306 log.cc:1196] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/9e2933d8d3f649e694564faa9808ea2f
I20260619 08:22:55.344798 23306 ts_tablet_manager.cc:1939] T 9e2933d8d3f649e694564faa9808ea2f P 0531976adc1c464aa31b1e9125aa6e92: Deleting consensus metadata
I20260619 08:22:55.345199 22792 catalog_manager.cc:4841] TS 0531976adc1c464aa31b1e9125aa6e92 (127.4.151.130:39659): tablet 9e2933d8d3f649e694564faa9808ea2f (table test-table [id=290077bac21c48d98310fd2f11395072]) successfully deleted
I20260619 08:22:55.340762 23305 tablet_replica.cc:330] T 9e2933d8d3f649e694564faa9808ea2f P 2d2b3c0711a8470c93333621fe28a176: stopping tablet replica
I20260619 08:22:55.345533 23305 raft_consensus.cc:2229] T 9e2933d8d3f649e694564faa9808ea2f P 2d2b3c0711a8470c93333621fe28a176 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:22:55.345649 23305 raft_consensus.cc:2258] T 9e2933d8d3f649e694564faa9808ea2f P 2d2b3c0711a8470c93333621fe28a176 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:22:55.346172 23305 ts_tablet_manager.cc:1905] T 9e2933d8d3f649e694564faa9808ea2f P 2d2b3c0711a8470c93333621fe28a176: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:22:55.347692 23305 ts_tablet_manager.cc:1918] T 9e2933d8d3f649e694564faa9808ea2f P 2d2b3c0711a8470c93333621fe28a176: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:22:55.347783 23305 log.cc:1196] T 9e2933d8d3f649e694564faa9808ea2f P 2d2b3c0711a8470c93333621fe28a176: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/9e2933d8d3f649e694564faa9808ea2f
I20260619 08:22:55.348031 23305 ts_tablet_manager.cc:1939] T 9e2933d8d3f649e694564faa9808ea2f P 2d2b3c0711a8470c93333621fe28a176: Deleting consensus metadata
I20260619 08:22:55.348453 22790 catalog_manager.cc:4841] TS 2d2b3c0711a8470c93333621fe28a176 (127.4.151.131:43651): tablet 9e2933d8d3f649e694564faa9808ea2f (table test-table [id=290077bac21c48d98310fd2f11395072]) successfully deleted
W20260619 08:22:55.348793 22791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.151.129:42749 (error 108)
I20260619 08:22:55.348973 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 23001
I20260619 08:22:55.349916 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.349785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:42749 (local address 127.0.0.1:47932)
0619 08:22:55.349837 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:55.349873 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.129:42749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 08:22:55.350028 22791 catalog_manager.cc:4584] TS f34059f2aa7c47e4ad7283097b0628ce (127.4.151.129:42749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e2933d8d3f649e694564faa9808ea2f: Network error: Client connection negotiation failed: client connection to 127.4.151.129:42749: connect: Connection refused (error 111)
I20260619 08:22:55.353415 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 23135
I20260619 08:22:55.359658 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 22773
2026-06-19T08:22:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2245 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T08:22:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:22:55Z Disabled control of system clock
I20260619 08:22:55.387338 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:44957
--webserver_interface=127.4.151.190
--webserver_port=0
--builtin_ntp_servers=127.4.151.148:45501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:44957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 08:22:55.469437 23311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:55.469655 23311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:55.469676 23311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:55.470893 23311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:22:55.470940 23311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:55.470953 23311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:55.470968 23311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:22:55.470979 23311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:22:55.472347 23311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:45501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:44957
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:44957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:55.472613 23311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:55.472934 23311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:55.477871 23319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:55.477878 23316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:55.477878 23317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:55.478186 23311 server_base.cc:1034] running on GCE node
I20260619 08:22:55.478479 23311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:55.478776 23311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:55.479972 23311 hybrid_clock.cc:648] HybridClock initialized: now 1781857375479945 us; error 58 us; skew 500 ppm
I20260619 08:22:55.487028 23311 webserver.cc:460] Webserver started at http://127.4.151.190:46599/ using document root <none> and password file <none>
I20260619 08:22:55.487458 23311 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:55.487545 23311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:55.487738 23311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:22:55.488991 23311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "b00d6b1b8b044f2eb070f07f414f0b1c"
format_stamp: "Formatted at 2026-06-19 08:22:55 on dist-test-slave-d3n8"
server_key: "39484fa42596ade79974ff3914162116"
server_key_iv: "f94487391fd11867823cb9a46c316635"
server_key_version: "encryptionkey@0"
I20260619 08:22:55.489471 23311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "b00d6b1b8b044f2eb070f07f414f0b1c"
format_stamp: "Formatted at 2026-06-19 08:22:55 on dist-test-slave-d3n8"
server_key: "39484fa42596ade79974ff3914162116"
server_key_iv: "f94487391fd11867823cb9a46c316635"
server_key_version: "encryptionkey@0"
I20260619 08:22:55.490962 23311 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:22:55.491926 23325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:55.492134 23311 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:22:55.492245 23311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "b00d6b1b8b044f2eb070f07f414f0b1c"
format_stamp: "Formatted at 2026-06-19 08:22:55 on dist-test-slave-d3n8"
server_key: "39484fa42596ade79974ff3914162116"
server_key_iv: "f94487391fd11867823cb9a46c316635"
server_key_version: "encryptionkey@0"
I20260619 08:22:55.492341 23311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:55.507009 23311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:55.507386 23311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:55.507560 23311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:55.511860 23311 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:44957
I20260619 08:22:55.511897 23377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:44957 every 8 connection(s)
I20260619 08:22:55.512404 23311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:22:55.513484 23378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:55.515883 23378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b00d6b1b8b044f2eb070f07f414f0b1c: Bootstrap starting.
I20260619 08:22:55.516558 23378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b00d6b1b8b044f2eb070f07f414f0b1c: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:55.516885 23378 log.cc:824] T 00000000000000000000000000000000 P b00d6b1b8b044f2eb070f07f414f0b1c: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:55.517702 23378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b00d6b1b8b044f2eb070f07f414f0b1c: No bootstrap required, opened a new log
I20260619 08:22:55.519358 23378 raft_consensus.cc:348] T 00000000000000000000000000000000 P b00d6b1b8b044f2eb070f07f414f0b1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00d6b1b8b044f2eb070f07f414f0b1c" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44957 } }
I20260619 08:22:55.519501 23378 raft_consensus.cc:374] T 00000000000000000000000000000000 P b00d6b1b8b044f2eb070f07f414f0b1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:55.519523 23378 raft_consensus.cc:729] T 00000000000000000000000000000000 P b00d6b1b8b044f2eb070f07f414f0b1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00d6b1b8b044f2eb070f07f414f0b1c, State: Initialized, Role: FOLLOWER
I20260619 08:22:55.519709 23378 consensus_queue.cc:260] T 00000000000000000000000000000000 P b00d6b1b8b044f2eb070f07f414f0b1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00d6b1b8b044f2eb070f07f414f0b1c" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44957 } }
I20260619 08:22:55.519779 23378 raft_consensus.cc:388] T 00000000000000000000000000000000 P b00d6b1b8b044f2eb070f07f414f0b1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:22:55.519824 23378 raft_consensus.cc:482] T 00000000000000000000000000000000 P b00d6b1b8b044f2eb070f07f414f0b1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:22:55.519876 23378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b00d6b1b8b044f2eb070f07f414f0b1c [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:55.520486 23378 raft_consensus.cc:504] T 00000000000000000000000000000000 P b00d6b1b8b044f2eb070f07f414f0b1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00d6b1b8b044f2eb070f07f414f0b1c" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44957 } }
I20260619 08:22:55.520602 23378 leader_election.cc:302] T 00000000000000000000000000000000 P b00d6b1b8b044f2eb070f07f414f0b1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b00d6b1b8b044f2eb070f07f414f0b1c; no voters:
I20260619 08:22:55.520808 23378 leader_election.cc:288] T 00000000000000000000000000000000 P b00d6b1b8b044f2eb070f07f414f0b1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:22:55.520885 23381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b00d6b1b8b044f2eb070f07f414f0b1c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:22:55.521064 23381 raft_consensus.cc:686] T 00000000000000000000000000000000 P b00d6b1b8b044f2eb070f07f414f0b1c [term 1 LEADER]: Becoming Leader. State: Replica: b00d6b1b8b044f2eb070f07f414f0b1c, State: Running, Role: LEADER
I20260619 08:22:55.521098 23378 sys_catalog.cc:564] T 00000000000000000000000000000000 P b00d6b1b8b044f2eb070f07f414f0b1c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:22:55.521184 23381 consensus_queue.cc:237] T 00000000000000000000000000000000 P b00d6b1b8b044f2eb070f07f414f0b1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00d6b1b8b044f2eb070f07f414f0b1c" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44957 } }
I20260619 08:22:55.521600 23383 sys_catalog.cc:455] T 00000000000000000000000000000000 P b00d6b1b8b044f2eb070f07f414f0b1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b00d6b1b8b044f2eb070f07f414f0b1c. Latest consensus state: current_term: 1 leader_uuid: "b00d6b1b8b044f2eb070f07f414f0b1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00d6b1b8b044f2eb070f07f414f0b1c" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44957 } } }
I20260619 08:22:55.521688 23382 sys_catalog.cc:455] T 00000000000000000000000000000000 P b00d6b1b8b044f2eb070f07f414f0b1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b00d6b1b8b044f2eb070f07f414f0b1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00d6b1b8b044f2eb070f07f414f0b1c" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44957 } } }
I20260619 08:22:55.521701 23383 sys_catalog.cc:458] T 00000000000000000000000000000000 P b00d6b1b8b044f2eb070f07f414f0b1c [sys.catalog]: This master's current role is: LEADER
I20260619 08:22:55.521764 23382 sys_catalog.cc:458] T 00000000000000000000000000000000 P b00d6b1b8b044f2eb070f07f414f0b1c [sys.catalog]: This master's current role is: LEADER
I20260619 08:22:55.521943 23390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:22:55.522084 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 23311
I20260619 08:22:55.522182 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:22:55.522410 4702 external_mini_cluster.cc:1428] Setting key 1362658e0fbc87cdb35ed5133e3c0b3c
I20260619 08:22:55.522598 23390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:22:55.524550 23390 catalog_manager.cc:1347] Generated new cluster ID: 91ac46d01d87485ebfedb3a79292a0a2
I20260619 08:22:55.524593 23390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:22:55.530180 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.523072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36196 (local address 127.4.151.190:44957)
0619 08:22:55.523671 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:55.523680 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:55.523710 (+ 30us) server_negotiation.cc:407] Connection header received
0619 08:22:55.523822 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:55.523828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:55.523904 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:55.524061 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:55.524404 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.526720 (+ 2316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.527811 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.528480 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.528616 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.529422 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:55.529426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:55.529426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:22:55.529643 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:55.529687 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:22:55.529703 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:55.529787 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:55.529872 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 08:22:55.529956 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":529,"thread_start_us":100,"threads_started":1}
I20260619 08:22:55.530455 23390 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:22:55.531034 23390 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:22:55.533437 23390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b00d6b1b8b044f2eb070f07f414f0b1c: Generated new TSK 0
I20260619 08:22:55.533604 23390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:22:55.535292 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:44957
--builtin_ntp_servers=127.4.151.148:45501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 08:22:55.624357 23402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:55.624591 23402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:55.624625 23402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:55.625792 23402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:55.625856 23402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:55.625897 23402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:22:55.627238 23402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:45501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:44957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:55.627512 23402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:55.627811 23402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:55.633126 23408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:55.633198 23410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:55.633241 23407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:55.633370 23402 server_base.cc:1034] running on GCE node
I20260619 08:22:55.633631 23402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:55.633926 23402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:55.635167 23402 hybrid_clock.cc:648] HybridClock initialized: now 1781857375635138 us; error 43 us; skew 500 ppm
I20260619 08:22:55.636574 23402 webserver.cc:460] Webserver started at http://127.4.151.129:33137/ using document root <none> and password file <none>
I20260619 08:22:55.636849 23402 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:55.636893 23402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:55.637028 23402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:22:55.638157 23402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "505e7c99c51c4fccb8a7539ffbb46e08"
format_stamp: "Formatted at 2026-06-19 08:22:55 on dist-test-slave-d3n8"
server_key: "9a55e932860879b3d70149fdab18fbdf"
server_key_iv: "cd7f7348f3655824dc2feec9aa191557"
server_key_version: "encryptionkey@0"
I20260619 08:22:55.638509 23402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "505e7c99c51c4fccb8a7539ffbb46e08"
format_stamp: "Formatted at 2026-06-19 08:22:55 on dist-test-slave-d3n8"
server_key: "9a55e932860879b3d70149fdab18fbdf"
server_key_iv: "cd7f7348f3655824dc2feec9aa191557"
server_key_version: "encryptionkey@0"
I20260619 08:22:55.639847 23402 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 08:22:55.640735 23416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:55.640904 23402 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:22:55.640982 23402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "505e7c99c51c4fccb8a7539ffbb46e08"
format_stamp: "Formatted at 2026-06-19 08:22:55 on dist-test-slave-d3n8"
server_key: "9a55e932860879b3d70149fdab18fbdf"
server_key_iv: "cd7f7348f3655824dc2feec9aa191557"
server_key_version: "encryptionkey@0"
I20260619 08:22:55.641072 23402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:55.650547 23402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:55.650840 23402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:55.650992 23402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:55.651214 23402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:22:55.651614 23402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:22:55.651669 23402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:55.651706 23402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:22:55.651729 23402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:55.658777 23402 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:35547
I20260619 08:22:55.658819 23529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:35547 every 8 connection(s)
I20260619 08:22:55.659160 23402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
I20260619 08:22:55.664367 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.659906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:39895 (local address 127.4.151.190:44957)
0619 08:22:55.659978 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:55.659980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:55.660186 (+ 206us) server_negotiation.cc:407] Connection header received
0619 08:22:55.660669 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:55.660669 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:55.660678 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:55.660751 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:55.661423 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.661921 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.662945 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.663132 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.663188 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.664145 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:55.664145 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:55.664146 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:22:55.664150 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:55.664160 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:22:55.664163 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:55.664200 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:55.664266 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 08:22:55.664288 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 08:22:55.664537 23532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.659875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44957 (local address 127.4.151.129:39895)
0619 08:22:55.660053 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:55.660094 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:55.660586 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:55.660805 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:55.660808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:55.660816 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:22:55.661311 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:55.661315 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.662028 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.662029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:55.662857 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:55.662860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.663238 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.663239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:55.663370 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.664014 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:22:55.664015 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:22:55.664043 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:22:55.664066 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:22:55.664202 (+ 136us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:22:55.664204 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:22:55.664206 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:22:55.664253 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 08:22:55.664327 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260619 08:22:55.664564 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 23402
I20260619 08:22:55.664659 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:22:55.664871 4702 external_mini_cluster.cc:1428] Setting key b07fc318ac225399fd2b63d78132d1f5
I20260619 08:22:55.665257 23530 heartbeater.cc:347] Connected to a master server at 127.4.151.190:44957
I20260619 08:22:55.665357 23530 heartbeater.cc:464] Registering TS with master...
I20260619 08:22:55.665580 23530 heartbeater.cc:511] Master 127.4.151.190:44957 requested a full tablet report, sending...
I20260619 08:22:55.666029 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:44957
--builtin_ntp_servers=127.4.151.148:45501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 08:22:55.666109 23342 ts_manager.cc:194] Registered new tserver with Master: 505e7c99c51c4fccb8a7539ffbb46e08 (127.4.151.129:35547)
I20260619 08:22:55.666725 23342 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.151.129:39895
W20260619 08:22:55.748046 23533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:55.748289 23533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:55.748313 23533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:55.749593 23533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:55.749637 23533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:55.749663 23533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:22:55.751062 23533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:45501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:44957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:55.751340 23533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:55.751663 23533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:55.757185 23539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:55.757229 23538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:55.757282 23541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:55.757349 23533 server_base.cc:1034] running on GCE node
I20260619 08:22:55.757679 23533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:55.757928 23533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:55.759099 23533 hybrid_clock.cc:648] HybridClock initialized: now 1781857375759082 us; error 51 us; skew 500 ppm
I20260619 08:22:55.760501 23533 webserver.cc:460] Webserver started at http://127.4.151.130:44673/ using document root <none> and password file <none>
I20260619 08:22:55.760780 23533 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:55.760830 23533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:55.760968 23533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:22:55.762151 23533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "8aab6db8068e4cbaa94d607c28c3d918"
format_stamp: "Formatted at 2026-06-19 08:22:55 on dist-test-slave-d3n8"
server_key: "1ddf0b10729b6fed5d845c2f2ff714d7"
server_key_iv: "d7d0908cf2a2df9933198502f5d4c37f"
server_key_version: "encryptionkey@0"
I20260619 08:22:55.762526 23533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "8aab6db8068e4cbaa94d607c28c3d918"
format_stamp: "Formatted at 2026-06-19 08:22:55 on dist-test-slave-d3n8"
server_key: "1ddf0b10729b6fed5d845c2f2ff714d7"
server_key_iv: "d7d0908cf2a2df9933198502f5d4c37f"
server_key_version: "encryptionkey@0"
I20260619 08:22:55.763955 23533 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:22:55.764771 23547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:55.764966 23533 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:22:55.765061 23533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "8aab6db8068e4cbaa94d607c28c3d918"
format_stamp: "Formatted at 2026-06-19 08:22:55 on dist-test-slave-d3n8"
server_key: "1ddf0b10729b6fed5d845c2f2ff714d7"
server_key_iv: "d7d0908cf2a2df9933198502f5d4c37f"
server_key_version: "encryptionkey@0"
I20260619 08:22:55.765122 23533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:55.773844 23533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:55.774127 23533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:55.774231 23533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:55.774410 23533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:22:55.774673 23533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:22:55.774716 23533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:55.774739 23533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:22:55.774752 23533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:55.781090 23533 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:37943
I20260619 08:22:55.781147 23660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:37943 every 8 connection(s)
I20260619 08:22:55.781541 23533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
I20260619 08:22:55.787303 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.782331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:47947 (local address 127.4.151.190:44957)
0619 08:22:55.782404 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:55.782407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:55.782661 (+ 254us) server_negotiation.cc:407] Connection header received
0619 08:22:55.783316 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:55.783317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:55.783328 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:55.783387 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:55.784062 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.784577 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.785625 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.785863 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.785941 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.787020 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:55.787021 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:55.787021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:22:55.787026 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:55.787056 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:22:55.787061 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:55.787126 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:55.787178 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 08:22:55.787202 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 08:22:55.787459 23663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.782320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44957 (local address 127.4.151.130:47947)
0619 08:22:55.782539 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:55.782574 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:55.783058 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:55.783463 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:55.783466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:55.783474 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:22:55.783973 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:55.783977 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.784656 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.784657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:55.785549 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:55.785551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.785966 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.785968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:55.786140 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.786848 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:22:55.786853 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:22:55.786912 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:22:55.786942 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:22:55.787129 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:22:55.787131 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:22:55.787133 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:22:55.787179 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 08:22:55.787262 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
I20260619 08:22:55.787905 23661 heartbeater.cc:347] Connected to a master server at 127.4.151.190:44957
I20260619 08:22:55.787985 23661 heartbeater.cc:464] Registering TS with master...
I20260619 08:22:55.788178 23661 heartbeater.cc:511] Master 127.4.151.190:44957 requested a full tablet report, sending...
I20260619 08:22:55.788560 23342 ts_manager.cc:194] Registered new tserver with Master: 8aab6db8068e4cbaa94d607c28c3d918 (127.4.151.130:37943)
I20260619 08:22:55.788959 23342 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.151.130:47947
I20260619 08:22:55.790953 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 23533
I20260619 08:22:55.791042 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:22:55.791242 4702 external_mini_cluster.cc:1428] Setting key 37f5213a58b145c777ae760505dd3efd
I20260619 08:22:55.792086 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:44957
--builtin_ntp_servers=127.4.151.148:45501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 08:22:55.874564 23664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:55.874811 23664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:55.874848 23664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:55.876044 23664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:55.876118 23664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:55.876163 23664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:22:55.877568 23664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:45501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:44957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:55.877857 23664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:55.878180 23664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:55.883734 23670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:55.883734 23664 server_base.cc:1034] running on GCE node
W20260619 08:22:55.883757 23669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:55.883754 23672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:55.884260 23664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:55.884567 23664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:55.885778 23664 hybrid_clock.cc:648] HybridClock initialized: now 1781857375885762 us; error 47 us; skew 500 ppm
I20260619 08:22:55.887141 23664 webserver.cc:460] Webserver started at http://127.4.151.131:42467/ using document root <none> and password file <none>
I20260619 08:22:55.887424 23664 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:55.887501 23664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:55.887728 23664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:22:55.888962 23664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "8902fae8e0bb4524a8b02089d3e9f3db"
format_stamp: "Formatted at 2026-06-19 08:22:55 on dist-test-slave-d3n8"
server_key: "10e6a187d9d24378121b6a6e3c975c21"
server_key_iv: "199a076858faf1b5f1309c13a9baa5a3"
server_key_version: "encryptionkey@0"
I20260619 08:22:55.889358 23664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "8902fae8e0bb4524a8b02089d3e9f3db"
format_stamp: "Formatted at 2026-06-19 08:22:55 on dist-test-slave-d3n8"
server_key: "10e6a187d9d24378121b6a6e3c975c21"
server_key_iv: "199a076858faf1b5f1309c13a9baa5a3"
server_key_version: "encryptionkey@0"
I20260619 08:22:55.890838 23664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:22:55.891616 23678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:55.891819 23664 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:22:55.891927 23664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "8902fae8e0bb4524a8b02089d3e9f3db"
format_stamp: "Formatted at 2026-06-19 08:22:55 on dist-test-slave-d3n8"
server_key: "10e6a187d9d24378121b6a6e3c975c21"
server_key_iv: "199a076858faf1b5f1309c13a9baa5a3"
server_key_version: "encryptionkey@0"
I20260619 08:22:55.892021 23664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:55.906955 23664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:55.907320 23664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:55.907488 23664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:55.907733 23664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:22:55.908095 23664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:22:55.908152 23664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:55.908190 23664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:22:55.908218 23664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:55.914008 23664 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:42025
I20260619 08:22:55.914099 23791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:42025 every 8 connection(s)
I20260619 08:22:55.914462 23664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
I20260619 08:22:55.917230 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 23664
I20260619 08:22:55.917358 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:22:55.917587 4702 external_mini_cluster.cc:1428] Setting key 3acc8badf3f869523831404416bd760b
I20260619 08:22:55.920846 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.915294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:47509 (local address 127.4.151.190:44957)
0619 08:22:55.915365 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:55.915367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:55.915679 (+ 312us) server_negotiation.cc:407] Connection header received
0619 08:22:55.916227 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:55.916228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:55.916241 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:55.916299 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:55.917021 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.917830 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.919290 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.919484 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.919543 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.920411 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:55.920412 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:55.920412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:22:55.920416 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:55.920549 (+ 133us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:22:55.920555 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:55.920616 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:55.920724 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 08:22:55.920764 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 08:22:55.921087 23794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.915367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:44957 (local address 127.4.151.131:47509)
0619 08:22:55.915551 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:55.915586 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:55.916109 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:55.916378 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:55.916382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:55.916390 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:22:55.916919 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:55.916923 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.917994 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.917995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:55.919191 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:55.919194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.919566 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.919566 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:55.919708 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.920264 (+ 556us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:22:55.920264 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:22:55.920295 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:22:55.920320 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:22:55.920621 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:22:55.920627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:22:55.920631 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:22:55.920726 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 08:22:55.920834 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
I20260619 08:22:55.921581 23792 heartbeater.cc:347] Connected to a master server at 127.4.151.190:44957
I20260619 08:22:55.921703 23792 heartbeater.cc:464] Registering TS with master...
I20260619 08:22:55.921875 23792 heartbeater.cc:511] Master 127.4.151.190:44957 requested a full tablet report, sending...
I20260619 08:22:55.922257 23342 ts_manager.cc:194] Registered new tserver with Master: 8902fae8e0bb4524a8b02089d3e9f3db (127.4.151.131:42025)
I20260619 08:22:55.922708 23342 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.151.131:47509
I20260619 08:22:55.929877 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:22:55.934542 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.931513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36198 (local address 127.4.151.190:44957)
0619 08:22:55.931583 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:55.931585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:55.931592 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:22:55.931613 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:55.931614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:55.931628 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:55.931688 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:55.931881 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.932372 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.933207 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.933472 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.933530 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.934276 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:55.934277 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:55.934277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:22:55.934282 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:55.934297 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:22:55.934301 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:55.934381 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:55.934468 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 08:22:55.934480 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 08:22:55.936619 23342 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:22:55.938542 23342 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:22:55.945561 23804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.941282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47682 (local address 127.4.151.130:37943)
0619 08:22:55.941488 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:55.941493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:55.941903 (+ 410us) server_negotiation.cc:407] Connection header received
0619 08:22:55.941945 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:55.941947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:55.941963 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:55.942032 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:55.942216 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.942907 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.944433 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.945057 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.945139 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.945231 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:55.945437 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 08:22:55.945478 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":110,"threads_started":1}
I20260619 08:22:55.945577 23803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.941109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45328 (local address 127.4.151.129:35547)
0619 08:22:55.941517 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:55.941521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:55.941533 (+ 12us) server_negotiation.cc:407] Connection header received
0619 08:22:55.941576 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:55.941578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:55.941594 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:55.941685 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:55.941888 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.943103 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.944404 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.945132 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.945206 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.945293 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:55.945452 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 08:22:55.945497 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":177,"threads_started":1}
I20260619 08:22:55.945812 23800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.940955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:42025 (local address 127.0.0.1:42916)
0619 08:22:55.941092 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:55.941132 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:55.941199 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:55.941767 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:55.941768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:55.941770 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:55.941802 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:55.941803 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.943218 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.943219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:55.944293 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:55.944296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.945239 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.945240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:55.945312 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.945317 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:22:55.945362 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 08:22:55.945462 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":3,"thread_start_us":84,"threads_started":1}
I20260619 08:22:55.945894 23802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.940986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:35547 (local address 127.0.0.1:45328)
0619 08:22:55.941420 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:55.941432 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:55.941486 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:55.941713 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:55.941716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:55.941719 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:55.941793 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:55.941796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.943201 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.943202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:55.944316 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:55.944318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.945281 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.945281 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:55.945335 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.945338 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:22:55.945431 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 08:22:55.945480 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":8,"thread_start_us":98,"threads_started":1}
I20260619 08:22:55.945894 23801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.941178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:37943 (local address 127.0.0.1:47682)
0619 08:22:55.941787 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:55.941797 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:55.941844 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:55.942093 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:55.942093 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:55.942095 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:55.942139 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:55.942140 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.942992 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.942992 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:55.944315 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:55.944318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.945138 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.945139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:55.945219 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.945233 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 08:22:55.945300 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 08:22:55.945355 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":4,"thread_start_us":77,"threads_started":1}
I20260619 08:22:55.946684 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:55.941332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42916 (local address 127.4.151.131:42025)
0619 08:22:55.941550 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:55.941553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:55.941562 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:22:55.941597 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:55.941599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:55.941613 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:55.941698 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:55.941888 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.943125 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.944398 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:55.945132 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:55.945508 (+ 376us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:55.945596 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:55.946569 (+ 973us) server_negotiation.cc:299] Negotiation successful
0619 08:22:55.946605 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":115,"threads_started":1}
I20260619 08:22:55.946658 23595 tablet_service.cc:1467] Processing CreateTablet for tablet c32876f5a9ba435abe9c1d019cc9a8d5 (DEFAULT_TABLE table=test-table [id=6242a00024524c4eaa58584792cfe4a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:22:55.946656 23464 tablet_service.cc:1467] Processing CreateTablet for tablet c32876f5a9ba435abe9c1d019cc9a8d5 (DEFAULT_TABLE table=test-table [id=6242a00024524c4eaa58584792cfe4a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:22:55.947005 23464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32876f5a9ba435abe9c1d019cc9a8d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:55.947005 23595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32876f5a9ba435abe9c1d019cc9a8d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:55.947279 23726 tablet_service.cc:1467] Processing CreateTablet for tablet c32876f5a9ba435abe9c1d019cc9a8d5 (DEFAULT_TABLE table=test-table [id=6242a00024524c4eaa58584792cfe4a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:22:55.947566 23726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32876f5a9ba435abe9c1d019cc9a8d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:55.950289 23806 tablet_bootstrap.cc:492] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8902fae8e0bb4524a8b02089d3e9f3db: Bootstrap starting.
I20260619 08:22:55.950579 23807 tablet_bootstrap.cc:492] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918: Bootstrap starting.
I20260619 08:22:55.950991 23806 tablet_bootstrap.cc:654] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8902fae8e0bb4524a8b02089d3e9f3db: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:55.951174 23807 tablet_bootstrap.cc:654] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:55.951339 23806 log.cc:824] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8902fae8e0bb4524a8b02089d3e9f3db: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:55.951486 23807 log.cc:824] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:55.951854 23808 tablet_bootstrap.cc:492] T c32876f5a9ba435abe9c1d019cc9a8d5 P 505e7c99c51c4fccb8a7539ffbb46e08: Bootstrap starting.
I20260619 08:22:55.952088 23806 tablet_bootstrap.cc:492] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8902fae8e0bb4524a8b02089d3e9f3db: No bootstrap required, opened a new log
I20260619 08:22:55.952173 23806 ts_tablet_manager.cc:1397] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8902fae8e0bb4524a8b02089d3e9f3db: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:22:55.952471 23808 tablet_bootstrap.cc:654] T c32876f5a9ba435abe9c1d019cc9a8d5 P 505e7c99c51c4fccb8a7539ffbb46e08: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:55.952818 23808 log.cc:824] T c32876f5a9ba435abe9c1d019cc9a8d5 P 505e7c99c51c4fccb8a7539ffbb46e08: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:55.953413 23807 tablet_bootstrap.cc:492] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918: No bootstrap required, opened a new log
I20260619 08:22:55.953485 23807 ts_tablet_manager.cc:1397] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 08:22:55.953617 23808 tablet_bootstrap.cc:492] T c32876f5a9ba435abe9c1d019cc9a8d5 P 505e7c99c51c4fccb8a7539ffbb46e08: No bootstrap required, opened a new log
I20260619 08:22:55.953701 23808 ts_tablet_manager.cc:1397] T c32876f5a9ba435abe9c1d019cc9a8d5 P 505e7c99c51c4fccb8a7539ffbb46e08: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:22:55.955114 23806 raft_consensus.cc:348] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8902fae8e0bb4524a8b02089d3e9f3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8902fae8e0bb4524a8b02089d3e9f3db" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42025 } } peers { permanent_uuid: "8aab6db8068e4cbaa94d607c28c3d918" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37943 } } peers { permanent_uuid: "505e7c99c51c4fccb8a7539ffbb46e08" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 35547 } }
I20260619 08:22:55.955320 23806 raft_consensus.cc:374] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8902fae8e0bb4524a8b02089d3e9f3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:55.955358 23806 raft_consensus.cc:729] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8902fae8e0bb4524a8b02089d3e9f3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8902fae8e0bb4524a8b02089d3e9f3db, State: Initialized, Role: FOLLOWER
I20260619 08:22:55.955425 23807 raft_consensus.cc:348] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8902fae8e0bb4524a8b02089d3e9f3db" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42025 } } peers { permanent_uuid: "8aab6db8068e4cbaa94d607c28c3d918" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37943 } } peers { permanent_uuid: "505e7c99c51c4fccb8a7539ffbb46e08" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 35547 } }
I20260619 08:22:55.955506 23806 consensus_queue.cc:260] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8902fae8e0bb4524a8b02089d3e9f3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8902fae8e0bb4524a8b02089d3e9f3db" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42025 } } peers { permanent_uuid: "8aab6db8068e4cbaa94d607c28c3d918" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37943 } } peers { permanent_uuid: "505e7c99c51c4fccb8a7539ffbb46e08" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 35547 } }
I20260619 08:22:55.955605 23807 raft_consensus.cc:374] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:55.955649 23807 raft_consensus.cc:729] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aab6db8068e4cbaa94d607c28c3d918, State: Initialized, Role: FOLLOWER
I20260619 08:22:55.955790 23806 ts_tablet_manager.cc:1428] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8902fae8e0bb4524a8b02089d3e9f3db: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 08:22:55.955770 23807 consensus_queue.cc:260] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8902fae8e0bb4524a8b02089d3e9f3db" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42025 } } peers { permanent_uuid: "8aab6db8068e4cbaa94d607c28c3d918" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37943 } } peers { permanent_uuid: "505e7c99c51c4fccb8a7539ffbb46e08" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 35547 } }
I20260619 08:22:55.955922 23792 heartbeater.cc:503] Master 127.4.151.190:44957 was elected leader, sending a full tablet report...
I20260619 08:22:55.956079 23807 ts_tablet_manager.cc:1428] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:22:55.956207 23661 heartbeater.cc:503] Master 127.4.151.190:44957 was elected leader, sending a full tablet report...
I20260619 08:22:55.956631 23808 raft_consensus.cc:348] T c32876f5a9ba435abe9c1d019cc9a8d5 P 505e7c99c51c4fccb8a7539ffbb46e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8902fae8e0bb4524a8b02089d3e9f3db" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42025 } } peers { permanent_uuid: "8aab6db8068e4cbaa94d607c28c3d918" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37943 } } peers { permanent_uuid: "505e7c99c51c4fccb8a7539ffbb46e08" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 35547 } }
I20260619 08:22:55.956830 23808 raft_consensus.cc:374] T c32876f5a9ba435abe9c1d019cc9a8d5 P 505e7c99c51c4fccb8a7539ffbb46e08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:55.956868 23808 raft_consensus.cc:729] T c32876f5a9ba435abe9c1d019cc9a8d5 P 505e7c99c51c4fccb8a7539ffbb46e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 505e7c99c51c4fccb8a7539ffbb46e08, State: Initialized, Role: FOLLOWER
I20260619 08:22:55.956993 23808 consensus_queue.cc:260] T c32876f5a9ba435abe9c1d019cc9a8d5 P 505e7c99c51c4fccb8a7539ffbb46e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8902fae8e0bb4524a8b02089d3e9f3db" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42025 } } peers { permanent_uuid: "8aab6db8068e4cbaa94d607c28c3d918" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37943 } } peers { permanent_uuid: "505e7c99c51c4fccb8a7539ffbb46e08" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 35547 } }
I20260619 08:22:55.957258 23808 ts_tablet_manager.cc:1428] T c32876f5a9ba435abe9c1d019cc9a8d5 P 505e7c99c51c4fccb8a7539ffbb46e08: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 08:22:55.957299 23530 heartbeater.cc:503] Master 127.4.151.190:44957 was elected leader, sending a full tablet report...
W20260619 08:22:56.032147 23662 tablet.cc:2369] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:22:56.062711 23813 raft_consensus.cc:482] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:22:56.062880 23813 raft_consensus.cc:504] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8902fae8e0bb4524a8b02089d3e9f3db" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42025 } } peers { permanent_uuid: "8aab6db8068e4cbaa94d607c28c3d918" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37943 } } peers { permanent_uuid: "505e7c99c51c4fccb8a7539ffbb46e08" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 35547 } }
I20260619 08:22:56.063339 23813 leader_election.cc:288] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8902fae8e0bb4524a8b02089d3e9f3db (127.4.151.131:42025), 505e7c99c51c4fccb8a7539ffbb46e08 (127.4.151.129:35547)
I20260619 08:22:56.067370 23663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:56.063524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:42025 (local address 127.4.151.130:59411)
0619 08:22:56.063580 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:56.063592 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:56.063640 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:56.063856 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:56.063857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:56.063859 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:56.063935 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:56.063937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:56.064887 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:56.064888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:56.066240 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:56.066244 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:56.067055 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:56.067056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:56.067153 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:56.067158 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:22:56.067223 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 08:22:56.067278 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":64}
I20260619 08:22:56.067376 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:56.063592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:59411 (local address 127.4.151.131:42025)
0619 08:22:56.063634 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:56.063636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:56.063647 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:22:56.063758 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:56.063759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:56.063771 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:56.063827 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:56.064028 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:56.064798 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:56.066363 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:56.066962 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:56.067026 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:56.067033 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:56.067239 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 08:22:56.067304 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:22:56.067411 23815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:56.063544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:35547 (local address 127.4.151.130:54351)
0619 08:22:56.063707 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:56.063724 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:56.063778 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:56.063890 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:56.063890 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:56.063891 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:56.063944 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:56.063945 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:56.064887 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:56.064888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:56.066284 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:56.066286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:56.067057 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:56.067058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:56.067230 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:56.067233 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:22:56.067282 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 08:22:56.067309 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":80,"thread_start_us":93,"threads_started":1}
I20260619 08:22:56.067442 23803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:56.063652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:54351 (local address 127.4.151.129:35547)
0619 08:22:56.063708 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:56.063711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:56.063789 (+ 78us) server_negotiation.cc:407] Connection header received
0619 08:22:56.063819 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:56.063820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:56.063832 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:56.063895 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:56.064039 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:56.064774 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:56.066367 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:56.066928 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:56.067016 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:56.067026 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:56.067328 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 08:22:56.067369 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 08:22:56.067763 23484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32876f5a9ba435abe9c1d019cc9a8d5" candidate_uuid: "8aab6db8068e4cbaa94d607c28c3d918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "505e7c99c51c4fccb8a7539ffbb46e08" is_pre_election: true
I20260619 08:22:56.067795 23746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32876f5a9ba435abe9c1d019cc9a8d5" candidate_uuid: "8aab6db8068e4cbaa94d607c28c3d918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8902fae8e0bb4524a8b02089d3e9f3db" is_pre_election: true
I20260619 08:22:56.067924 23484 raft_consensus.cc:2454] T c32876f5a9ba435abe9c1d019cc9a8d5 P 505e7c99c51c4fccb8a7539ffbb46e08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8aab6db8068e4cbaa94d607c28c3d918 in term 0.
I20260619 08:22:56.067950 23746 raft_consensus.cc:2454] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8902fae8e0bb4524a8b02089d3e9f3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8aab6db8068e4cbaa94d607c28c3d918 in term 0.
I20260619 08:22:56.068210 23551 leader_election.cc:302] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8902fae8e0bb4524a8b02089d3e9f3db, 8aab6db8068e4cbaa94d607c28c3d918; no voters:
I20260619 08:22:56.068420 23813 raft_consensus.cc:2781] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:22:56.068512 23813 raft_consensus.cc:482] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:22:56.068549 23813 raft_consensus.cc:3037] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:56.069289 23813 raft_consensus.cc:504] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8902fae8e0bb4524a8b02089d3e9f3db" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42025 } } peers { permanent_uuid: "8aab6db8068e4cbaa94d607c28c3d918" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37943 } } peers { permanent_uuid: "505e7c99c51c4fccb8a7539ffbb46e08" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 35547 } }
I20260619 08:22:56.069479 23813 leader_election.cc:288] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918 [CANDIDATE]: Term 1 election: Requested vote from peers 8902fae8e0bb4524a8b02089d3e9f3db (127.4.151.131:42025), 505e7c99c51c4fccb8a7539ffbb46e08 (127.4.151.129:35547)
I20260619 08:22:56.069614 23746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32876f5a9ba435abe9c1d019cc9a8d5" candidate_uuid: "8aab6db8068e4cbaa94d607c28c3d918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8902fae8e0bb4524a8b02089d3e9f3db"
I20260619 08:22:56.069641 23484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32876f5a9ba435abe9c1d019cc9a8d5" candidate_uuid: "8aab6db8068e4cbaa94d607c28c3d918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "505e7c99c51c4fccb8a7539ffbb46e08"
I20260619 08:22:56.069721 23746 raft_consensus.cc:3037] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8902fae8e0bb4524a8b02089d3e9f3db [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:56.069721 23484 raft_consensus.cc:3037] T c32876f5a9ba435abe9c1d019cc9a8d5 P 505e7c99c51c4fccb8a7539ffbb46e08 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:56.070468 23484 raft_consensus.cc:2454] T c32876f5a9ba435abe9c1d019cc9a8d5 P 505e7c99c51c4fccb8a7539ffbb46e08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8aab6db8068e4cbaa94d607c28c3d918 in term 1.
I20260619 08:22:56.070477 23746 raft_consensus.cc:2454] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8902fae8e0bb4524a8b02089d3e9f3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8aab6db8068e4cbaa94d607c28c3d918 in term 1.
I20260619 08:22:56.070644 23551 leader_election.cc:302] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8902fae8e0bb4524a8b02089d3e9f3db, 8aab6db8068e4cbaa94d607c28c3d918; no voters:
I20260619 08:22:56.070744 23813 raft_consensus.cc:2781] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:22:56.070914 23813 raft_consensus.cc:686] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918 [term 1 LEADER]: Becoming Leader. State: Replica: 8aab6db8068e4cbaa94d607c28c3d918, State: Running, Role: LEADER
I20260619 08:22:56.071022 23813 consensus_queue.cc:237] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8902fae8e0bb4524a8b02089d3e9f3db" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42025 } } peers { permanent_uuid: "8aab6db8068e4cbaa94d607c28c3d918" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37943 } } peers { permanent_uuid: "505e7c99c51c4fccb8a7539ffbb46e08" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 35547 } }
I20260619 08:22:56.071908 23341 catalog_manager.cc:5495] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8aab6db8068e4cbaa94d607c28c3d918 (127.4.151.130). New cstate: current_term: 1 leader_uuid: "8aab6db8068e4cbaa94d607c28c3d918" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8902fae8e0bb4524a8b02089d3e9f3db" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8aab6db8068e4cbaa94d607c28c3d918" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 37943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "505e7c99c51c4fccb8a7539ffbb46e08" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 35547 } health_report { overall_health: UNKNOWN } } }
I20260619 08:22:56.092242 23804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:56.089762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47688 (local address 127.4.151.130:37943)
0619 08:22:56.089826 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:56.089828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:56.089837 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:22:56.089879 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:56.089882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:56.089915 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:56.089996 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:22:56.090152 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:56.090620 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:56.091423 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:56.091617 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:56.091715 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:56.091883 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:22:56.091980 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:22:56.092092 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:56.092122 (+ 30us) server_negotiation.cc:299] Negotiation successful
0619 08:22:56.092130 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 08:22:56.094324 23746 raft_consensus.cc:1261] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8902fae8e0bb4524a8b02089d3e9f3db [term 1 FOLLOWER]: Refusing update from remote peer 8aab6db8068e4cbaa94d607c28c3d918: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:22:56.094341 23484 raft_consensus.cc:1261] T c32876f5a9ba435abe9c1d019cc9a8d5 P 505e7c99c51c4fccb8a7539ffbb46e08 [term 1 FOLLOWER]: Refusing update from remote peer 8aab6db8068e4cbaa94d607c28c3d918: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:22:56.094758 23813 consensus_queue.cc:1035] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8902fae8e0bb4524a8b02089d3e9f3db" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:22:56.094909 23816 consensus_queue.cc:1035] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918 [LEADER]: Connected to new peer: Peer: permanent_uuid: "505e7c99c51c4fccb8a7539ffbb46e08" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:22:56.097440 23819 mvcc.cc:205] Tried to move back new op lower bound from 7298487812479393792 to 7298487812387500032. Current Snapshot: MvccSnapshot[applied={T|T < 7298487812479393792}]
I20260619 08:22:56.098405 23820 mvcc.cc:205] Tried to move back new op lower bound from 7298487812479393792 to 7298487812387500032. Current Snapshot: MvccSnapshot[applied={T|T < 7298487812479393792}]
I20260619 08:22:56.099680 23342 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:22:56.099779 23342 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:22:56.100553 23342 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b00d6b1b8b044f2eb070f07f414f0b1c: Sending DeleteTablet for 3 replicas of tablet c32876f5a9ba435abe9c1d019cc9a8d5
I20260619 08:22:56.100890 23595 tablet_service.cc:1514] Processing DeleteTablet for tablet c32876f5a9ba435abe9c1d019cc9a8d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:22:56 UTC) from {username='slave'} at 127.0.0.1:47682
I20260619 08:22:56.100925 23726 tablet_service.cc:1514] Processing DeleteTablet for tablet c32876f5a9ba435abe9c1d019cc9a8d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:22:56 UTC) from {username='slave'} at 127.0.0.1:42916
I20260619 08:22:56.101138 23464 tablet_service.cc:1514] Processing DeleteTablet for tablet c32876f5a9ba435abe9c1d019cc9a8d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:22:56 UTC) from {username='slave'} at 127.0.0.1:45328
I20260619 08:22:56.101215 23829 tablet_replica.cc:330] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918: stopping tablet replica
I20260619 08:22:56.101323 23829 raft_consensus.cc:2229] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:22:56.101455 23829 raft_consensus.cc:2258] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:22:56.101696 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 23402
I20260619 08:22:56.102489 23829 ts_tablet_manager.cc:1905] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:22:56.104518 23829 ts_tablet_manager.cc:1918] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:22:56.104595 23829 log.cc:1196] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/c32876f5a9ba435abe9c1d019cc9a8d5
I20260619 08:22:56.104842 23829 ts_tablet_manager.cc:1939] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8aab6db8068e4cbaa94d607c28c3d918: Deleting consensus metadata
I20260619 08:22:56.105397 23329 catalog_manager.cc:4841] TS 8aab6db8068e4cbaa94d607c28c3d918 (127.4.151.130:37943): tablet c32876f5a9ba435abe9c1d019cc9a8d5 (table test-table [id=6242a00024524c4eaa58584792cfe4a0]) successfully deleted
I20260619 08:22:56.105733 23830 tablet_replica.cc:330] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8902fae8e0bb4524a8b02089d3e9f3db: stopping tablet replica
I20260619 08:22:56.105849 23830 raft_consensus.cc:2229] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8902fae8e0bb4524a8b02089d3e9f3db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:22:56.105958 23830 raft_consensus.cc:2258] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8902fae8e0bb4524a8b02089d3e9f3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:22:56.106382 23830 ts_tablet_manager.cc:1905] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8902fae8e0bb4524a8b02089d3e9f3db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:22:56.107637 23830 ts_tablet_manager.cc:1918] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8902fae8e0bb4524a8b02089d3e9f3db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:22:56.107728 23830 log.cc:1196] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8902fae8e0bb4524a8b02089d3e9f3db: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/c32876f5a9ba435abe9c1d019cc9a8d5
I20260619 08:22:56.107961 23830 ts_tablet_manager.cc:1939] T c32876f5a9ba435abe9c1d019cc9a8d5 P 8902fae8e0bb4524a8b02089d3e9f3db: Deleting consensus metadata
I20260619 08:22:56.108371 23329 catalog_manager.cc:4841] TS 8902fae8e0bb4524a8b02089d3e9f3db (127.4.151.131:42025): tablet c32876f5a9ba435abe9c1d019cc9a8d5 (table test-table [id=6242a00024524c4eaa58584792cfe4a0]) successfully deleted
W20260619 08:22:56.110034 23328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.151.129:35547 (error 108)
I20260619 08:22:56.110257 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 23533
I20260619 08:22:56.110860 23802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:56.110414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:35547 (local address 127.0.0.1:45342)
0619 08:22:56.110782 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:56.110829 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.129:35547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345}
W20260619 08:22:56.111006 23328 catalog_manager.cc:4584] TS 505e7c99c51c4fccb8a7539ffbb46e08 (127.4.151.129:35547): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c32876f5a9ba435abe9c1d019cc9a8d5: Network error: Client connection negotiation failed: client connection to 127.4.151.129:35547: connect: Connection refused (error 111)
I20260619 08:22:56.115156 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 23664
I20260619 08:22:56.119385 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 23311
2026-06-19T08:22:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (758 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:22:56 dist-test-slave-d3n8 krb5kdc[23835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:22:56 dist-test-slave-d3n8 krb5kdc[23835](info): set up 2 sockets
Jun 19 08:22:56 dist-test-slave-d3n8 krb5kdc[23835](info): commencing operation
krb5kdc: starting...
W20260619 08:22:57.162207 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:22:57 dist-test-slave-d3n8 krb5kdc[23835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:22:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:22:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:22:57.289188 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:39813
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:46851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:39813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:22:57.371009 23848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:57.371294 23848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:57.371340 23848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:57.372545 23848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:22:57.372619 23848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:57.372644 23848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:57.372674 23848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:22:57.372706 23848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:22:57.374375 23848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:46851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:39813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:39813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:57.374668 23848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:57.374908 23848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:57.380115 23856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:57.380090 23853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:57.380120 23848 server_base.cc:1034] running on GCE node
W20260619 08:22:57.380088 23854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:57.380555 23848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:57.380847 23848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:57.381990 23848 hybrid_clock.cc:648] HybridClock initialized: now 1781857377381972 us; error 40 us; skew 500 ppm
Jun 19 08:22:57 dist-test-slave-d3n8 krb5kdc[23835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857377, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:22:57.384825 23848 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:22:57.385255 23848 webserver.cc:460] Webserver started at http://127.4.151.190:36635/ using document root <none> and password file <none>
I20260619 08:22:57.385536 23848 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:57.385591 23848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:57.385690 23848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:22:57.386817 23848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "b96004452c1d4467a34e38dfc17f7b57"
format_stamp: "Formatted at 2026-06-19 08:22:57 on dist-test-slave-d3n8"
server_key: "13bf763ae08cf2e0a4e3c2527c77b81a"
server_key_iv: "3d24643d0d8db4fad270564c298e9a89"
server_key_version: "encryptionkey@0"
I20260619 08:22:57.387166 23848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "b96004452c1d4467a34e38dfc17f7b57"
format_stamp: "Formatted at 2026-06-19 08:22:57 on dist-test-slave-d3n8"
server_key: "13bf763ae08cf2e0a4e3c2527c77b81a"
server_key_iv: "3d24643d0d8db4fad270564c298e9a89"
server_key_version: "encryptionkey@0"
I20260619 08:22:57.388526 23848 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:22:57.389443 23863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:57.389649 23848 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:22:57.389761 23848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "b96004452c1d4467a34e38dfc17f7b57"
format_stamp: "Formatted at 2026-06-19 08:22:57 on dist-test-slave-d3n8"
server_key: "13bf763ae08cf2e0a4e3c2527c77b81a"
server_key_iv: "3d24643d0d8db4fad270564c298e9a89"
server_key_version: "encryptionkey@0"
I20260619 08:22:57.389865 23848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:57.403926 23848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:57.406528 23848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:57.406728 23848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:57.411077 23848 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:39813
I20260619 08:22:57.411135 23915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:39813 every 8 connection(s)
I20260619 08:22:57.411525 23848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:22:57.412482 23916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:57.414223 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 23848
I20260619 08:22:57.414358 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:22:57.414582 4702 external_mini_cluster.cc:1428] Setting key 39955c10caa6d8ca8ec9e878565d9230
I20260619 08:22:57.415442 23916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b96004452c1d4467a34e38dfc17f7b57: Bootstrap starting.
I20260619 08:22:57.416033 23916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b96004452c1d4467a34e38dfc17f7b57: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:57.416358 23916 log.cc:824] T 00000000000000000000000000000000 P b96004452c1d4467a34e38dfc17f7b57: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:57.417128 23916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b96004452c1d4467a34e38dfc17f7b57: No bootstrap required, opened a new log
I20260619 08:22:57.419353 23916 raft_consensus.cc:348] T 00000000000000000000000000000000 P b96004452c1d4467a34e38dfc17f7b57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96004452c1d4467a34e38dfc17f7b57" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 39813 } }
I20260619 08:22:57.419493 23916 raft_consensus.cc:374] T 00000000000000000000000000000000 P b96004452c1d4467a34e38dfc17f7b57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:57.419516 23916 raft_consensus.cc:729] T 00000000000000000000000000000000 P b96004452c1d4467a34e38dfc17f7b57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b96004452c1d4467a34e38dfc17f7b57, State: Initialized, Role: FOLLOWER
I20260619 08:22:57.419652 23916 consensus_queue.cc:260] T 00000000000000000000000000000000 P b96004452c1d4467a34e38dfc17f7b57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96004452c1d4467a34e38dfc17f7b57" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 39813 } }
I20260619 08:22:57.419727 23916 raft_consensus.cc:388] T 00000000000000000000000000000000 P b96004452c1d4467a34e38dfc17f7b57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:22:57.419770 23916 raft_consensus.cc:482] T 00000000000000000000000000000000 P b96004452c1d4467a34e38dfc17f7b57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:22:57.419819 23916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b96004452c1d4467a34e38dfc17f7b57 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:57.420578 23916 raft_consensus.cc:504] T 00000000000000000000000000000000 P b96004452c1d4467a34e38dfc17f7b57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96004452c1d4467a34e38dfc17f7b57" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 39813 } }
I20260619 08:22:57.420696 23916 leader_election.cc:302] T 00000000000000000000000000000000 P b96004452c1d4467a34e38dfc17f7b57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b96004452c1d4467a34e38dfc17f7b57; no voters:
I20260619 08:22:57.420882 23916 leader_election.cc:288] T 00000000000000000000000000000000 P b96004452c1d4467a34e38dfc17f7b57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:22:57.420959 23921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b96004452c1d4467a34e38dfc17f7b57 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:22:57.421119 23921 raft_consensus.cc:686] T 00000000000000000000000000000000 P b96004452c1d4467a34e38dfc17f7b57 [term 1 LEADER]: Becoming Leader. State: Replica: b96004452c1d4467a34e38dfc17f7b57, State: Running, Role: LEADER
I20260619 08:22:57.421159 23916 sys_catalog.cc:564] T 00000000000000000000000000000000 P b96004452c1d4467a34e38dfc17f7b57 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:22:57.421205 23921 consensus_queue.cc:237] T 00000000000000000000000000000000 P b96004452c1d4467a34e38dfc17f7b57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96004452c1d4467a34e38dfc17f7b57" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 39813 } }
I20260619 08:22:57.421594 23921 sys_catalog.cc:455] T 00000000000000000000000000000000 P b96004452c1d4467a34e38dfc17f7b57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b96004452c1d4467a34e38dfc17f7b57. Latest consensus state: current_term: 1 leader_uuid: "b96004452c1d4467a34e38dfc17f7b57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96004452c1d4467a34e38dfc17f7b57" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 39813 } } }
I20260619 08:22:57.421690 23921 sys_catalog.cc:458] T 00000000000000000000000000000000 P b96004452c1d4467a34e38dfc17f7b57 [sys.catalog]: This master's current role is: LEADER
I20260619 08:22:57.421605 23922 sys_catalog.cc:455] T 00000000000000000000000000000000 P b96004452c1d4467a34e38dfc17f7b57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b96004452c1d4467a34e38dfc17f7b57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96004452c1d4467a34e38dfc17f7b57" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 39813 } } }
I20260619 08:22:57.421869 23922 sys_catalog.cc:458] T 00000000000000000000000000000000 P b96004452c1d4467a34e38dfc17f7b57 [sys.catalog]: This master's current role is: LEADER
Jun 19 08:22:57 dist-test-slave-d3n8 krb5kdc[23835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:22:57.422536 23928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:22:57.423132 23928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:22:57.426174 23928 catalog_manager.cc:1347] Generated new cluster ID: 7d77cabc53ad49d283e8f3a4f16416d3
I20260619 08:22:57.426230 23928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:22:57.426913 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:57.415155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45928 (local address 127.4.151.190:39813)
0619 08:22:57.415349 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:57.415358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:57.415384 (+ 26us) server_negotiation.cc:407] Connection header received
0619 08:22:57.415945 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:57.415952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:57.416008 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:57.416186 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:57.416938 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:57.418021 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:57.419231 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:57.419789 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:57.419898 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:57.422928 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:57.422932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:57.422933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:22:57.422944 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:57.424931 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:57.425361 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:57.425364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:57.425364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:57.425500 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:57.425719 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:57.425720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:57.425720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:57.425963 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:57.426108 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:57.426545 (+ 437us) server_negotiation.cc:299] Negotiation successful
0619 08:22:57.426679 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
I20260619 08:22:57.431804 23928 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:22:57.432358 23928 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:22:57.442606 23928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b96004452c1d4467a34e38dfc17f7b57: Generated new TSK 0
I20260619 08:22:57.442833 23928 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.151.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
Entry for principal kudu/127.4.151.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab.
I20260619 08:22:57.477888 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:39813
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:46851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:22:57.588653 23942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:57.588935 23942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:57.588964 23942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:57.590459 23942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:57.590512 23942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:57.590548 23942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:22:57.592420 23942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:46851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.129.keytab
--principal=kudu/127.4.151.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:39813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:57.592746 23942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:57.593006 23942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:57.598730 23948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:57.598898 23950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:57.598922 23947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:57.599154 23942 server_base.cc:1034] running on GCE node
I20260619 08:22:57.599364 23942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:57.599599 23942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:57.600826 23942 hybrid_clock.cc:648] HybridClock initialized: now 1781857377600813 us; error 33 us; skew 500 ppm
Jun 19 08:22:57 dist-test-slave-d3n8 krb5kdc[23835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857377, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:22:57.604048 23942 init.cc:377] Logged in from keytab as kudu/127.4.151.129@KRBTEST.COM (short username kudu)
I20260619 08:22:57.604583 23942 webserver.cc:460] Webserver started at http://127.4.151.129:39779/ using document root <none> and password file <none>
I20260619 08:22:57.604856 23942 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:57.604925 23942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:57.605085 23942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:22:57.606245 23942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "64b1bb3533184c56b27141dd24748ce6"
format_stamp: "Formatted at 2026-06-19 08:22:57 on dist-test-slave-d3n8"
server_key: "884e8103da7208df37ceff6902283d64"
server_key_iv: "35062aaea1fc7f663eb9f408f6852b11"
server_key_version: "encryptionkey@0"
I20260619 08:22:57.606624 23942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "64b1bb3533184c56b27141dd24748ce6"
format_stamp: "Formatted at 2026-06-19 08:22:57 on dist-test-slave-d3n8"
server_key: "884e8103da7208df37ceff6902283d64"
server_key_iv: "35062aaea1fc7f663eb9f408f6852b11"
server_key_version: "encryptionkey@0"
I20260619 08:22:57.608003 23942 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 08:22:57.608913 23957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:57.609282 23942 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:22:57.609441 23942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "64b1bb3533184c56b27141dd24748ce6"
format_stamp: "Formatted at 2026-06-19 08:22:57 on dist-test-slave-d3n8"
server_key: "884e8103da7208df37ceff6902283d64"
server_key_iv: "35062aaea1fc7f663eb9f408f6852b11"
server_key_version: "encryptionkey@0"
I20260619 08:22:57.609562 23942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:57.622763 23942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:57.625108 23942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:57.625229 23942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:57.625627 23942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:22:57.625970 23942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:22:57.626005 23942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:57.626029 23942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:22:57.626077 23942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:57.632251 23942 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:34015
I20260619 08:22:57.632339 24070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:34015 every 8 connection(s)
I20260619 08:22:57.632709 23942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
Jun 19 08:22:57 dist-test-slave-d3n8 krb5kdc[23835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857377, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.129@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:22:57.642596 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:57.633534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:50621 (local address 127.4.151.190:39813)
0619 08:22:57.633606 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:57.633608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:57.633840 (+ 232us) server_negotiation.cc:407] Connection header received
0619 08:22:57.634345 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:57.634346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:57.634355 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:57.634407 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:57.635333 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:57.635842 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:57.636819 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:57.637011 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:57.637064 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:57.639575 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:57.639576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:57.639577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:22:57.639583 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:57.641088 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:57.641612 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:57.641612 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:57.641613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:57.641666 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:57.641958 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:57.641959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:57.641959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:57.642072 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:57.642122 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:57.642445 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 08:22:57.642526 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 08:22:57.642863 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 23942
I20260619 08:22:57.642966 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:22:57.642956 24073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:57.633516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:39813 (local address 127.4.151.129:50621)
0619 08:22:57.633708 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:57.633745 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:57.634254 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:57.634459 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:57.634462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:57.635038 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:22:57.635251 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:57.635256 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:57.635922 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:57.635923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:57.636743 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:57.636745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:57.637109 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:57.637110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:57.637262 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:57.637844 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:22:57.637852 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:22:57.639428 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:22:57.641206 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:57.641213 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:57.641215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:57.641543 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:57.641760 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:57.641760 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:57.641761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:57.641887 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:57.642157 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:22:57.642165 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:22:57.642308 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 08:22:57.642398 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 08:22:57.642531 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":88,"threads_started":1}
I20260619 08:22:57.643196 4702 external_mini_cluster.cc:1428] Setting key a264ab29f05822f51de4d5432802174e
I20260619 08:22:57.643759 24071 heartbeater.cc:347] Connected to a master server at 127.4.151.190:39813
I20260619 08:22:57.643868 24071 heartbeater.cc:464] Registering TS with master...
I20260619 08:22:57.644079 24071 heartbeater.cc:511] Master 127.4.151.190:39813 requested a full tablet report, sending...
I20260619 08:22:57.644658 23880 ts_manager.cc:194] Registered new tserver with Master: 64b1bb3533184c56b27141dd24748ce6 (127.4.151.129:34015)
I20260619 08:22:57.645334 23880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.129@KRBTEST.COM'} at 127.4.151.129:50621
WARNING: no policy specified for kudu/127.4.151.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
Entry for principal kudu/127.4.151.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab.
I20260619 08:22:57.669246 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:39813
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:46851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:22:57.753008 24076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:57.753235 24076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:57.753261 24076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:57.754581 24076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:57.754638 24076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:57.754669 24076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:22:57.756175 24076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:46851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.130.keytab
--principal=kudu/127.4.151.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:39813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:57.756397 24076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:57.756595 24076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:57.762102 24082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:57.762178 24084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:57.762264 24081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:57.762300 24076 server_base.cc:1034] running on GCE node
I20260619 08:22:57.762564 24076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:57.762866 24076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:57.764053 24076 hybrid_clock.cc:648] HybridClock initialized: now 1781857377764018 us; error 56 us; skew 500 ppm
Jun 19 08:22:57 dist-test-slave-d3n8 krb5kdc[23835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857377, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:22:57.767040 24076 init.cc:377] Logged in from keytab as kudu/127.4.151.130@KRBTEST.COM (short username kudu)
I20260619 08:22:57.767848 24076 webserver.cc:460] Webserver started at http://127.4.151.130:42031/ using document root <none> and password file <none>
I20260619 08:22:57.768180 24076 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:57.768270 24076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:57.768428 24076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:22:57.769654 24076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "86cf69aa05cb4703825e698317e2dc79"
format_stamp: "Formatted at 2026-06-19 08:22:57 on dist-test-slave-d3n8"
server_key: "d265c3e687d5a67bf1c24b33c549cbfc"
server_key_iv: "d32af2b43c1714a3a49031db78bb7e12"
server_key_version: "encryptionkey@0"
I20260619 08:22:57.770051 24076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "86cf69aa05cb4703825e698317e2dc79"
format_stamp: "Formatted at 2026-06-19 08:22:57 on dist-test-slave-d3n8"
server_key: "d265c3e687d5a67bf1c24b33c549cbfc"
server_key_iv: "d32af2b43c1714a3a49031db78bb7e12"
server_key_version: "encryptionkey@0"
I20260619 08:22:57.771544 24076 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:22:57.772423 24091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:57.772694 24076 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:22:57.772779 24076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "86cf69aa05cb4703825e698317e2dc79"
format_stamp: "Formatted at 2026-06-19 08:22:57 on dist-test-slave-d3n8"
server_key: "d265c3e687d5a67bf1c24b33c549cbfc"
server_key_iv: "d32af2b43c1714a3a49031db78bb7e12"
server_key_version: "encryptionkey@0"
I20260619 08:22:57.772837 24076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:57.785989 24076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:57.788558 24076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:57.788720 24076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:57.788926 24076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:22:57.789335 24076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:22:57.789420 24076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:57.789479 24076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:22:57.789518 24076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:57.795804 24076 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:44245
I20260619 08:22:57.795848 24204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:44245 every 8 connection(s)
I20260619 08:22:57.796195 24076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
Jun 19 08:22:57 dist-test-slave-d3n8 krb5kdc[23835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857377, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.130@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:22:57.804627 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 24076
I20260619 08:22:57.804774 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:22:57.805006 4702 external_mini_cluster.cc:1428] Setting key f84fe9ccadff8c51dbe86119ef63e1d6
I20260619 08:22:57.806937 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:57.797098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:33085 (local address 127.4.151.190:39813)
0619 08:22:57.797150 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:57.797152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:57.797254 (+ 102us) server_negotiation.cc:407] Connection header received
0619 08:22:57.797838 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:57.797840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:57.797851 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:57.797922 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:57.799025 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:57.799535 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:57.800663 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:57.800878 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:57.800951 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:57.803415 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:57.803417 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:57.803417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:22:57.803426 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:57.805187 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:57.805779 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:57.805779 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:57.805780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:57.805867 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:57.806163 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:57.806164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:57.806165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:57.806331 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:57.806401 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:57.806695 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 08:22:57.806791 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 08:22:57.806964 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:57.796933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:39813 (local address 127.4.151.130:33085)
0619 08:22:57.797095 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:57.797149 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:57.797717 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:57.798045 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:57.798050 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:57.798707 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:22:57.798918 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:57.798923 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:57.799637 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:57.799638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:57.800561 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:57.800563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:57.800994 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:57.800995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:57.801127 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:57.801756 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:22:57.801769 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:22:57.803299 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:22:57.805339 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:57.805350 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:57.805352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:57.805681 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:57.805960 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:57.805960 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:57.805961 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:57.806079 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:57.806402 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:22:57.806408 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:22:57.806537 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 08:22:57.806635 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 08:22:57.806784 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
I20260619 08:22:57.807512 24205 heartbeater.cc:347] Connected to a master server at 127.4.151.190:39813
I20260619 08:22:57.807641 24205 heartbeater.cc:464] Registering TS with master...
I20260619 08:22:57.807854 24205 heartbeater.cc:511] Master 127.4.151.190:39813 requested a full tablet report, sending...
I20260619 08:22:57.808300 23880 ts_manager.cc:194] Registered new tserver with Master: 86cf69aa05cb4703825e698317e2dc79 (127.4.151.130:44245)
I20260619 08:22:57.808737 23880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.130@KRBTEST.COM'} at 127.4.151.130:33085
WARNING: no policy specified for kudu/127.4.151.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
Entry for principal kudu/127.4.151.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab.
I20260619 08:22:57.833216 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:39813
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:46851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:22:57.922209 24210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:57.922463 24210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:57.922503 24210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:57.923825 24210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:57.923902 24210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:57.923949 24210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:22:57.925565 24210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:46851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.131.keytab
--principal=kudu/127.4.151.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:39813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:57.925849 24210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:57.926095 24210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:57.931663 24218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:57.931675 24215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:57.931675 24216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:57.931846 24210 server_base.cc:1034] running on GCE node
I20260619 08:22:57.932025 24210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:57.932294 24210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:57.933454 24210 hybrid_clock.cc:648] HybridClock initialized: now 1781857377933432 us; error 32 us; skew 500 ppm
Jun 19 08:22:57 dist-test-slave-d3n8 krb5kdc[23835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857377, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:22:57.935990 24210 init.cc:377] Logged in from keytab as kudu/127.4.151.131@KRBTEST.COM (short username kudu)
I20260619 08:22:57.936525 24210 webserver.cc:460] Webserver started at http://127.4.151.131:33141/ using document root <none> and password file <none>
I20260619 08:22:57.936770 24210 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:57.936841 24210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:57.936995 24210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:22:57.938200 24210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "178d795e854c4424b74fda23683612d4"
format_stamp: "Formatted at 2026-06-19 08:22:57 on dist-test-slave-d3n8"
server_key: "ba2602fb518e0491f3c956fe33e872b8"
server_key_iv: "912e57fa15c877c819d19df8ad303c50"
server_key_version: "encryptionkey@0"
I20260619 08:22:57.938572 24210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "178d795e854c4424b74fda23683612d4"
format_stamp: "Formatted at 2026-06-19 08:22:57 on dist-test-slave-d3n8"
server_key: "ba2602fb518e0491f3c956fe33e872b8"
server_key_iv: "912e57fa15c877c819d19df8ad303c50"
server_key_version: "encryptionkey@0"
I20260619 08:22:57.939990 24210 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 08:22:57.940871 24225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:57.941162 24210 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:22:57.941268 24210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "178d795e854c4424b74fda23683612d4"
format_stamp: "Formatted at 2026-06-19 08:22:57 on dist-test-slave-d3n8"
server_key: "ba2602fb518e0491f3c956fe33e872b8"
server_key_iv: "912e57fa15c877c819d19df8ad303c50"
server_key_version: "encryptionkey@0"
I20260619 08:22:57.941339 24210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:57.948787 24210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:57.951144 24210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:57.951326 24210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:57.951568 24210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:22:57.951882 24210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:22:57.951939 24210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:57.951983 24210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:22:57.952008 24210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:57.958182 24210 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:42249
I20260619 08:22:57.958271 24338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:42249 every 8 connection(s)
I20260619 08:22:57.958590 24210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
Jun 19 08:22:57 dist-test-slave-d3n8 krb5kdc[23835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857377, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.131@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:22:57.968189 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:57.959398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:45837 (local address 127.4.151.190:39813)
0619 08:22:57.959442 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:57.959445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:57.959653 (+ 208us) server_negotiation.cc:407] Connection header received
0619 08:22:57.960191 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:57.960192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:57.960205 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:57.960266 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:57.961230 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:57.961837 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:57.962839 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:57.963052 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:57.963144 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:57.965488 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:57.965489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:57.965490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:22:57.965497 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:57.966768 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:57.967177 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:57.967178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:57.967179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:57.967241 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:57.967505 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:57.967506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:57.967506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:57.967625 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:57.967703 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:57.967981 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 08:22:57.968064 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:22:57.968358 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:57.959358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:39813 (local address 127.4.151.131:45837)
0619 08:22:57.959531 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:57.959566 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:57.960080 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:57.960356 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:57.960360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:57.960928 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:22:57.961118 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:57.961123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:57.961933 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:57.961934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:57.962729 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:57.962731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:57.963156 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:57.963156 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:57.963295 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:57.963817 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:22:57.963826 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:22:57.965324 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:22:57.966864 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:57.966872 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:57.966874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:57.967096 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:57.967320 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:22:57.967321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:22:57.967321 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:22:57.967429 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:22:57.967693 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:22:57.967699 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:22:57.967844 (+ 145us) client_negotiation.cc:769] Sending connection context
0619 08:22:57.967959 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 08:22:57.968119 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260619 08:22:57.968797 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 24210
I20260619 08:22:57.968914 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:22:57.969095 4702 external_mini_cluster.cc:1428] Setting key 900c28d17ba42ebbd9e37cd419c25892
I20260619 08:22:57.969276 24339 heartbeater.cc:347] Connected to a master server at 127.4.151.190:39813
I20260619 08:22:57.969406 24339 heartbeater.cc:464] Registering TS with master...
I20260619 08:22:57.969687 24339 heartbeater.cc:511] Master 127.4.151.190:39813 requested a full tablet report, sending...
I20260619 08:22:57.970134 23880 ts_manager.cc:194] Registered new tserver with Master: 178d795e854c4424b74fda23683612d4 (127.4.151.131:42249)
I20260619 08:22:57.970572 23880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.151.131@KRBTEST.COM'} at 127.4.151.131:45837
I20260619 08:22:57.981290 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:22:57.989358 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:57.982825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45930 (local address 127.4.151.190:39813)
0619 08:22:57.982909 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:57.982911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:57.983029 (+ 118us) server_negotiation.cc:407] Connection header received
0619 08:22:57.983111 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:57.983112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:57.983129 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:57.983192 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:57.983807 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:57.984346 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:57.985114 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:57.985307 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:57.985433 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:57.986652 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:57.986654 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:57.986654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:22:57.986668 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:57.988098 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:57.988466 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:57.988468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:57.988468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:57.988526 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:22:57.988741 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:22:57.988742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:22:57.988742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:22:57.988907 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:57.989016 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:57.989159 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 08:22:57.989244 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 08:22:57.990967 23880 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:22:57.993170 23880 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:22:58.000516 24352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:57.996096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:42249 (local address 127.0.0.1:54068)
0619 08:22:57.996378 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:57.996423 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:57.996496 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:57.996857 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:57.996861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:57.996863 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:57.996946 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:57.996948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:57.997992 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:57.997993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:57.999126 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:57.999129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.000155 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.000156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:58.000253 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.000268 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 08:22:58.000346 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 08:22:58.000406 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":52,"threads_started":1}
I20260619 08:22:58.000638 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:57.996101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:44245 (local address 127.0.0.1:44362)
0619 08:22:57.996604 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:57.996615 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:57.996657 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:57.996896 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:57.996897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:57.996899 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:57.996970 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:57.996973 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:57.998526 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:57.998527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:57.999529 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:57.999531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.000347 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.000347 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:58.000409 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.000413 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:22:58.000473 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 08:22:58.000509 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":71,"thread_start_us":110,"threads_started":1}
I20260619 08:22:58.000818 24353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:57.996185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54068 (local address 127.4.151.131:42249)
0619 08:22:57.996644 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:57.996648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:57.996658 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:22:57.996708 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:57.996710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:57.996744 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:57.997028 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:57.997104 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:57.997887 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:57.999230 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.000051 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.000153 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.000221 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:58.000659 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 08:22:58.000718 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":159,"threads_started":1}
I20260619 08:22:58.000628 24355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:57.996467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44362 (local address 127.4.151.130:44245)
0619 08:22:57.996642 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:57.996648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:57.996660 (+ 12us) server_negotiation.cc:407] Connection header received
0619 08:22:57.996765 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:57.996767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:57.996785 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:57.997055 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:57.997137 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:57.998402 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:57.999613 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.000184 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.000321 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.000374 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:58.000500 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 08:22:58.000541 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":91,"threads_started":1}
I20260619 08:22:58.001400 24354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:57.996376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37066 (local address 127.4.151.129:34015)
0619 08:22:57.996817 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:57.996821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:57.996830 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:22:57.996872 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:57.996874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:57.996889 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:57.997107 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:57.997193 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:57.998446 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:57.999954 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.000980 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.001052 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.001105 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:58.001257 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 08:22:58.001298 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":88,"threads_started":1}
I20260619 08:22:58.001408 24351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:57.996101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:34015 (local address 127.0.0.1:37066)
0619 08:22:57.996383 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:57.996425 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:57.996496 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:57.996989 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:57.996989 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:57.996991 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:57.997030 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:57.997032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:57.998579 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:57.998580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:57.999855 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:57.999858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.001072 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.001073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:58.001157 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.001162 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:22:58.001237 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 08:22:58.001289 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"spinlock_wait_cycles":384,"thread_start_us":155,"threads_started":1}
I20260619 08:22:58.001591 24273 tablet_service.cc:1467] Processing CreateTablet for tablet ce0978a29f30483787944522d9260c26 (DEFAULT_TABLE table=test-table [id=fcc38cdf823742ce8a370dbcec9e9fbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:22:58.002004 24273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce0978a29f30483787944522d9260c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:58.002185 24005 tablet_service.cc:1467] Processing CreateTablet for tablet ce0978a29f30483787944522d9260c26 (DEFAULT_TABLE table=test-table [id=fcc38cdf823742ce8a370dbcec9e9fbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:22:58.002524 24005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce0978a29f30483787944522d9260c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:58.002453 24139 tablet_service.cc:1467] Processing CreateTablet for tablet ce0978a29f30483787944522d9260c26 (DEFAULT_TABLE table=test-table [id=fcc38cdf823742ce8a370dbcec9e9fbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:22:58.002774 24139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce0978a29f30483787944522d9260c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:58.004499 24356 tablet_bootstrap.cc:492] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4: Bootstrap starting.
I20260619 08:22:58.005137 24356 tablet_bootstrap.cc:654] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:58.005842 24356 log.cc:824] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:58.006682 24356 tablet_bootstrap.cc:492] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4: No bootstrap required, opened a new log
I20260619 08:22:58.006784 24356 ts_tablet_manager.cc:1397] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 08:22:58.008119 24356 raft_consensus.cc:348] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b1bb3533184c56b27141dd24748ce6" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 34015 } } peers { permanent_uuid: "178d795e854c4424b74fda23683612d4" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42249 } } peers { permanent_uuid: "86cf69aa05cb4703825e698317e2dc79" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 44245 } }
I20260619 08:22:58.008283 24356 raft_consensus.cc:374] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:58.008329 24356 raft_consensus.cc:729] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 178d795e854c4424b74fda23683612d4, State: Initialized, Role: FOLLOWER
I20260619 08:22:58.008469 24356 consensus_queue.cc:260] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b1bb3533184c56b27141dd24748ce6" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 34015 } } peers { permanent_uuid: "178d795e854c4424b74fda23683612d4" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42249 } } peers { permanent_uuid: "86cf69aa05cb4703825e698317e2dc79" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 44245 } }
I20260619 08:22:58.008776 24356 ts_tablet_manager.cc:1428] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 08:22:58.008781 24339 heartbeater.cc:503] Master 127.4.151.190:39813 was elected leader, sending a full tablet report...
I20260619 08:22:58.009009 24358 tablet_bootstrap.cc:492] T ce0978a29f30483787944522d9260c26 P 86cf69aa05cb4703825e698317e2dc79: Bootstrap starting.
I20260619 08:22:58.009783 24358 tablet_bootstrap.cc:654] T ce0978a29f30483787944522d9260c26 P 86cf69aa05cb4703825e698317e2dc79: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:58.010332 24358 log.cc:824] T ce0978a29f30483787944522d9260c26 P 86cf69aa05cb4703825e698317e2dc79: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:58.010592 24359 tablet_bootstrap.cc:492] T ce0978a29f30483787944522d9260c26 P 64b1bb3533184c56b27141dd24748ce6: Bootstrap starting.
I20260619 08:22:58.011176 24358 tablet_bootstrap.cc:492] T ce0978a29f30483787944522d9260c26 P 86cf69aa05cb4703825e698317e2dc79: No bootstrap required, opened a new log
I20260619 08:22:58.011276 24358 ts_tablet_manager.cc:1397] T ce0978a29f30483787944522d9260c26 P 86cf69aa05cb4703825e698317e2dc79: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 08:22:58.011328 24359 tablet_bootstrap.cc:654] T ce0978a29f30483787944522d9260c26 P 64b1bb3533184c56b27141dd24748ce6: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:58.011899 24359 log.cc:824] T ce0978a29f30483787944522d9260c26 P 64b1bb3533184c56b27141dd24748ce6: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:58.012677 24359 tablet_bootstrap.cc:492] T ce0978a29f30483787944522d9260c26 P 64b1bb3533184c56b27141dd24748ce6: No bootstrap required, opened a new log
I20260619 08:22:58.012760 24359 ts_tablet_manager.cc:1397] T ce0978a29f30483787944522d9260c26 P 64b1bb3533184c56b27141dd24748ce6: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260619 08:22:58.013523 24358 raft_consensus.cc:348] T ce0978a29f30483787944522d9260c26 P 86cf69aa05cb4703825e698317e2dc79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b1bb3533184c56b27141dd24748ce6" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 34015 } } peers { permanent_uuid: "178d795e854c4424b74fda23683612d4" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42249 } } peers { permanent_uuid: "86cf69aa05cb4703825e698317e2dc79" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 44245 } }
I20260619 08:22:58.013706 24358 raft_consensus.cc:374] T ce0978a29f30483787944522d9260c26 P 86cf69aa05cb4703825e698317e2dc79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:58.013746 24358 raft_consensus.cc:729] T ce0978a29f30483787944522d9260c26 P 86cf69aa05cb4703825e698317e2dc79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86cf69aa05cb4703825e698317e2dc79, State: Initialized, Role: FOLLOWER
I20260619 08:22:58.013862 24358 consensus_queue.cc:260] T ce0978a29f30483787944522d9260c26 P 86cf69aa05cb4703825e698317e2dc79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b1bb3533184c56b27141dd24748ce6" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 34015 } } peers { permanent_uuid: "178d795e854c4424b74fda23683612d4" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42249 } } peers { permanent_uuid: "86cf69aa05cb4703825e698317e2dc79" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 44245 } }
I20260619 08:22:58.014161 24358 ts_tablet_manager.cc:1428] T ce0978a29f30483787944522d9260c26 P 86cf69aa05cb4703825e698317e2dc79: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 08:22:58.014164 24205 heartbeater.cc:503] Master 127.4.151.190:39813 was elected leader, sending a full tablet report...
I20260619 08:22:58.015014 24359 raft_consensus.cc:348] T ce0978a29f30483787944522d9260c26 P 64b1bb3533184c56b27141dd24748ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b1bb3533184c56b27141dd24748ce6" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 34015 } } peers { permanent_uuid: "178d795e854c4424b74fda23683612d4" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42249 } } peers { permanent_uuid: "86cf69aa05cb4703825e698317e2dc79" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 44245 } }
I20260619 08:22:58.015175 24359 raft_consensus.cc:374] T ce0978a29f30483787944522d9260c26 P 64b1bb3533184c56b27141dd24748ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:58.015203 24359 raft_consensus.cc:729] T ce0978a29f30483787944522d9260c26 P 64b1bb3533184c56b27141dd24748ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64b1bb3533184c56b27141dd24748ce6, State: Initialized, Role: FOLLOWER
I20260619 08:22:58.015309 24359 consensus_queue.cc:260] T ce0978a29f30483787944522d9260c26 P 64b1bb3533184c56b27141dd24748ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b1bb3533184c56b27141dd24748ce6" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 34015 } } peers { permanent_uuid: "178d795e854c4424b74fda23683612d4" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42249 } } peers { permanent_uuid: "86cf69aa05cb4703825e698317e2dc79" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 44245 } }
I20260619 08:22:58.015575 24359 ts_tablet_manager.cc:1428] T ce0978a29f30483787944522d9260c26 P 64b1bb3533184c56b27141dd24748ce6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:22:58.015584 24071 heartbeater.cc:503] Master 127.4.151.190:39813 was elected leader, sending a full tablet report...
W20260619 08:22:58.046844 24206 tablet.cc:2369] T ce0978a29f30483787944522d9260c26 P 86cf69aa05cb4703825e698317e2dc79: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:22:58.133571 24072 tablet.cc:2369] T ce0978a29f30483787944522d9260c26 P 64b1bb3533184c56b27141dd24748ce6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:22:58.176342 24360 raft_consensus.cc:482] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:22:58.176512 24360 raft_consensus.cc:504] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b1bb3533184c56b27141dd24748ce6" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 34015 } } peers { permanent_uuid: "178d795e854c4424b74fda23683612d4" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42249 } } peers { permanent_uuid: "86cf69aa05cb4703825e698317e2dc79" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 44245 } }
I20260619 08:22:58.176963 24360 leader_election.cc:288] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64b1bb3533184c56b27141dd24748ce6 (127.4.151.129:34015), 86cf69aa05cb4703825e698317e2dc79 (127.4.151.130:44245)
I20260619 08:22:58.180373 24354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.177244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:59499 (local address 127.4.151.129:34015)
0619 08:22:58.177288 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:58.177290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:58.177670 (+ 380us) server_negotiation.cc:407] Connection header received
0619 08:22:58.177702 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:58.177704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:58.177714 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:58.177773 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:58.177916 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.178388 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.179494 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.180036 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.180099 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.180108 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:58.180232 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 08:22:58.180280 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:22:58.180416 24355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.177221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:56281 (local address 127.4.151.130:44245)
0619 08:22:58.177269 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:58.177272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:58.177310 (+ 38us) server_negotiation.cc:407] Connection header received
0619 08:22:58.177428 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:58.177429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:58.177438 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:58.177519 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:58.177817 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.178353 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.179400 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.179975 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.180056 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.180066 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:58.180304 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 08:22:58.180350 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 08:22:58.180397 24365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.177163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:34015 (local address 127.4.151.131:59499)
0619 08:22:58.177587 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:58.177596 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:58.177655 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:58.177777 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:58.177778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:58.177779 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:58.177838 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:58.177839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.178486 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.178487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:58.179341 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:58.179343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.180103 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.180103 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:58.180160 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.180162 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:22:58.180213 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 08:22:58.180269 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":266,"thread_start_us":116,"threads_started":1}
I20260619 08:22:58.180420 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.177142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:44245 (local address 127.4.151.131:56281)
0619 08:22:58.177211 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:58.177224 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:58.177324 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:58.177527 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:58.177528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:58.177530 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:58.177611 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:58.177612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.178444 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.178445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:58.179326 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:58.179328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.180055 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.180055 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:58.180153 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.180158 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:22:58.180214 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 08:22:58.180274 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":45}
I20260619 08:22:58.180742 24021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0978a29f30483787944522d9260c26" candidate_uuid: "178d795e854c4424b74fda23683612d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64b1bb3533184c56b27141dd24748ce6" is_pre_election: true
I20260619 08:22:58.180804 24159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0978a29f30483787944522d9260c26" candidate_uuid: "178d795e854c4424b74fda23683612d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86cf69aa05cb4703825e698317e2dc79" is_pre_election: true
I20260619 08:22:58.180902 24021 raft_consensus.cc:2454] T ce0978a29f30483787944522d9260c26 P 64b1bb3533184c56b27141dd24748ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 178d795e854c4424b74fda23683612d4 in term 0.
I20260619 08:22:58.180968 24159 raft_consensus.cc:2454] T ce0978a29f30483787944522d9260c26 P 86cf69aa05cb4703825e698317e2dc79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 178d795e854c4424b74fda23683612d4 in term 0.
I20260619 08:22:58.181137 24227 leader_election.cc:302] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 178d795e854c4424b74fda23683612d4, 64b1bb3533184c56b27141dd24748ce6; no voters:
I20260619 08:22:58.181316 24360 raft_consensus.cc:2781] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:22:58.181394 24360 raft_consensus.cc:482] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:22:58.181424 24360 raft_consensus.cc:3037] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:58.182220 24360 raft_consensus.cc:504] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b1bb3533184c56b27141dd24748ce6" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 34015 } } peers { permanent_uuid: "178d795e854c4424b74fda23683612d4" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42249 } } peers { permanent_uuid: "86cf69aa05cb4703825e698317e2dc79" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 44245 } }
I20260619 08:22:58.182363 24360 leader_election.cc:288] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4 [CANDIDATE]: Term 1 election: Requested vote from peers 64b1bb3533184c56b27141dd24748ce6 (127.4.151.129:34015), 86cf69aa05cb4703825e698317e2dc79 (127.4.151.130:44245)
I20260619 08:22:58.182507 24021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0978a29f30483787944522d9260c26" candidate_uuid: "178d795e854c4424b74fda23683612d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64b1bb3533184c56b27141dd24748ce6"
I20260619 08:22:58.182588 24021 raft_consensus.cc:3037] T ce0978a29f30483787944522d9260c26 P 64b1bb3533184c56b27141dd24748ce6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:58.182556 24159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0978a29f30483787944522d9260c26" candidate_uuid: "178d795e854c4424b74fda23683612d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86cf69aa05cb4703825e698317e2dc79"
I20260619 08:22:58.182660 24159 raft_consensus.cc:3037] T ce0978a29f30483787944522d9260c26 P 86cf69aa05cb4703825e698317e2dc79 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:58.183362 24021 raft_consensus.cc:2454] T ce0978a29f30483787944522d9260c26 P 64b1bb3533184c56b27141dd24748ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 178d795e854c4424b74fda23683612d4 in term 1.
I20260619 08:22:58.183522 24227 leader_election.cc:302] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 178d795e854c4424b74fda23683612d4, 64b1bb3533184c56b27141dd24748ce6; no voters:
I20260619 08:22:58.183600 24360 raft_consensus.cc:2781] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:22:58.183691 24159 raft_consensus.cc:2454] T ce0978a29f30483787944522d9260c26 P 86cf69aa05cb4703825e698317e2dc79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 178d795e854c4424b74fda23683612d4 in term 1.
I20260619 08:22:58.183765 24360 raft_consensus.cc:686] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4 [term 1 LEADER]: Becoming Leader. State: Replica: 178d795e854c4424b74fda23683612d4, State: Running, Role: LEADER
I20260619 08:22:58.183863 24360 consensus_queue.cc:237] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b1bb3533184c56b27141dd24748ce6" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 34015 } } peers { permanent_uuid: "178d795e854c4424b74fda23683612d4" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42249 } } peers { permanent_uuid: "86cf69aa05cb4703825e698317e2dc79" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 44245 } }
I20260619 08:22:58.184695 23880 catalog_manager.cc:5495] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 178d795e854c4424b74fda23683612d4 (127.4.151.131). New cstate: current_term: 1 leader_uuid: "178d795e854c4424b74fda23683612d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b1bb3533184c56b27141dd24748ce6" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 34015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "178d795e854c4424b74fda23683612d4" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 42249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86cf69aa05cb4703825e698317e2dc79" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 44245 } health_report { overall_health: UNKNOWN } } }
W20260619 08:22:58.209218 24340 tablet.cc:2369] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:22:58.230208 24353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.227226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54072 (local address 127.4.151.131:42249)
0619 08:22:58.227290 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:58.227292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:58.227466 (+ 174us) server_negotiation.cc:407] Connection header received
0619 08:22:58.227548 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:58.227549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:58.227572 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:58.227642 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:22:58.227859 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.228493 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.229297 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.229605 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.229670 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.229856 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:22:58.229970 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:22:58.230069 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:58.230088 (+ 19us) server_negotiation.cc:299] Negotiation successful
0619 08:22:58.230109 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 08:22:58.232115 24159 raft_consensus.cc:1261] T ce0978a29f30483787944522d9260c26 P 86cf69aa05cb4703825e698317e2dc79 [term 1 FOLLOWER]: Refusing update from remote peer 178d795e854c4424b74fda23683612d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:22:58.232199 24021 raft_consensus.cc:1261] T ce0978a29f30483787944522d9260c26 P 64b1bb3533184c56b27141dd24748ce6 [term 1 FOLLOWER]: Refusing update from remote peer 178d795e854c4424b74fda23683612d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:22:58.232494 24366 consensus_queue.cc:1035] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86cf69aa05cb4703825e698317e2dc79" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:22:58.232625 24360 consensus_queue.cc:1035] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64b1bb3533184c56b27141dd24748ce6" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:22:58.235495 24371 mvcc.cc:205] Tried to move back new op lower bound from 7298487821235736576 to 7298487821041741824. Current Snapshot: MvccSnapshot[applied={T|T < 7298487821235736576}]
I20260619 08:22:58.235845 24370 mvcc.cc:205] Tried to move back new op lower bound from 7298487821235736576 to 7298487821041741824. Current Snapshot: MvccSnapshot[applied={T|T < 7298487821235736576}]
I20260619 08:22:58.237481 23880 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:22:58.237587 23880 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:22:58.238435 23880 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b96004452c1d4467a34e38dfc17f7b57: Sending DeleteTablet for 3 replicas of tablet ce0978a29f30483787944522d9260c26
I20260619 08:22:58.238821 24139 tablet_service.cc:1514] Processing DeleteTablet for tablet ce0978a29f30483787944522d9260c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:22:58 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:44362
I20260619 08:22:58.238821 24273 tablet_service.cc:1514] Processing DeleteTablet for tablet ce0978a29f30483787944522d9260c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:22:58 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:54068
I20260619 08:22:58.239056 24005 tablet_service.cc:1514] Processing DeleteTablet for tablet ce0978a29f30483787944522d9260c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:22:58 UTC) from {username='kudu', principal='kudu/127.4.151.190@KRBTEST.COM'} at 127.0.0.1:37066
I20260619 08:22:58.239073 24378 tablet_replica.cc:330] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4: stopping tablet replica
I20260619 08:22:58.239169 24378 raft_consensus.cc:2229] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:22:58.239259 24378 raft_consensus.cc:2258] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:22:58.239324 24380 tablet_replica.cc:330] T ce0978a29f30483787944522d9260c26 P 64b1bb3533184c56b27141dd24748ce6: stopping tablet replica
I20260619 08:22:58.239419 24380 raft_consensus.cc:2229] T ce0978a29f30483787944522d9260c26 P 64b1bb3533184c56b27141dd24748ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:22:58.239535 24380 raft_consensus.cc:2258] T ce0978a29f30483787944522d9260c26 P 64b1bb3533184c56b27141dd24748ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:22:58.239567 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 23942
I20260619 08:22:58.239712 24378 ts_tablet_manager.cc:1905] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:22:58.241466 24378 ts_tablet_manager.cc:1918] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:22:58.241532 24378 log.cc:1196] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/ce0978a29f30483787944522d9260c26
I20260619 08:22:58.241822 24378 ts_tablet_manager.cc:1939] T ce0978a29f30483787944522d9260c26 P 178d795e854c4424b74fda23683612d4: Deleting consensus metadata
I20260619 08:22:58.241993 24379 tablet_replica.cc:330] T ce0978a29f30483787944522d9260c26 P 86cf69aa05cb4703825e698317e2dc79: stopping tablet replica
I20260619 08:22:58.242092 24379 raft_consensus.cc:2229] T ce0978a29f30483787944522d9260c26 P 86cf69aa05cb4703825e698317e2dc79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:22:58.242204 24379 raft_consensus.cc:2258] T ce0978a29f30483787944522d9260c26 P 86cf69aa05cb4703825e698317e2dc79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:22:58.242666 24379 ts_tablet_manager.cc:1905] T ce0978a29f30483787944522d9260c26 P 86cf69aa05cb4703825e698317e2dc79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:22:58.243041 23866 catalog_manager.cc:4841] TS 178d795e854c4424b74fda23683612d4 (127.4.151.131:42249): tablet ce0978a29f30483787944522d9260c26 (table test-table [id=fcc38cdf823742ce8a370dbcec9e9fbf]) successfully deleted
I20260619 08:22:58.244428 24379 ts_tablet_manager.cc:1918] T ce0978a29f30483787944522d9260c26 P 86cf69aa05cb4703825e698317e2dc79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:22:58.244508 24379 log.cc:1196] T ce0978a29f30483787944522d9260c26 P 86cf69aa05cb4703825e698317e2dc79: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/ce0978a29f30483787944522d9260c26
I20260619 08:22:58.244810 24379 ts_tablet_manager.cc:1939] T ce0978a29f30483787944522d9260c26 P 86cf69aa05cb4703825e698317e2dc79: Deleting consensus metadata
I20260619 08:22:58.245451 23867 catalog_manager.cc:4841] TS 86cf69aa05cb4703825e698317e2dc79 (127.4.151.130:44245): tablet ce0978a29f30483787944522d9260c26 (table test-table [id=fcc38cdf823742ce8a370dbcec9e9fbf]) successfully deleted
W20260619 08:22:58.246254 23865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.151.129:34015 (error 108)
I20260619 08:22:58.246459 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 24076
I20260619 08:22:58.247205 24351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.247082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:34015 (local address 127.0.0.1:37082)
0619 08:22:58.247131 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:58.247169 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.129:34015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 08:22:58.247313 23865 catalog_manager.cc:4584] TS 64b1bb3533184c56b27141dd24748ce6 (127.4.151.129:34015): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce0978a29f30483787944522d9260c26: Network error: Client connection negotiation failed: client connection to 127.4.151.129:34015: connect: Connection refused (error 111)
I20260619 08:22:58.252131 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 24210
I20260619 08:22:58.256836 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 23848
2026-06-19T08:22:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2138 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T08:22:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:22:58Z Disabled control of system clock
I20260619 08:22:58.284835 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:40751
--webserver_interface=127.4.151.190
--webserver_port=0
--builtin_ntp_servers=127.4.151.148:36605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:40751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 08:22:58.367323 24385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:58.367555 24385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:58.367578 24385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:58.368731 24385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:22:58.368772 24385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:58.368785 24385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:58.368798 24385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:22:58.368809 24385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:22:58.370224 24385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:36605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:40751
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:40751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:58.370445 24385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:58.370694 24385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:58.376050 24390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:58.376073 24393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:58.376068 24391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:58.376333 24385 server_base.cc:1034] running on GCE node
I20260619 08:22:58.376690 24385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:58.377012 24385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:58.378192 24385 hybrid_clock.cc:648] HybridClock initialized: now 1781857378378150 us; error 57 us; skew 500 ppm
I20260619 08:22:58.379657 24385 webserver.cc:460] Webserver started at http://127.4.151.190:38535/ using document root <none> and password file <none>
I20260619 08:22:58.379930 24385 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:58.380004 24385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:58.380157 24385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:22:58.381224 24385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "ed7d3c6543eb4fa094c938caf47eb4e4"
format_stamp: "Formatted at 2026-06-19 08:22:58 on dist-test-slave-d3n8"
server_key: "d62aa756a25ea2cf8d34b6286add45f5"
server_key_iv: "dd2cc5a32bdefd84e9db994256a159b6"
server_key_version: "encryptionkey@0"
I20260619 08:22:58.381614 24385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "ed7d3c6543eb4fa094c938caf47eb4e4"
format_stamp: "Formatted at 2026-06-19 08:22:58 on dist-test-slave-d3n8"
server_key: "d62aa756a25ea2cf8d34b6286add45f5"
server_key_iv: "dd2cc5a32bdefd84e9db994256a159b6"
server_key_version: "encryptionkey@0"
I20260619 08:22:58.382931 24385 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:22:58.383773 24399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:58.384024 24385 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:22:58.384140 24385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "ed7d3c6543eb4fa094c938caf47eb4e4"
format_stamp: "Formatted at 2026-06-19 08:22:58 on dist-test-slave-d3n8"
server_key: "d62aa756a25ea2cf8d34b6286add45f5"
server_key_iv: "dd2cc5a32bdefd84e9db994256a159b6"
server_key_version: "encryptionkey@0"
I20260619 08:22:58.384280 24385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:58.402240 24385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:58.402590 24385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:58.402755 24385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:58.406847 24385 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:40751
I20260619 08:22:58.406896 24451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:40751 every 8 connection(s)
I20260619 08:22:58.407272 24385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:22:58.408215 24452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:58.409371 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 24385
I20260619 08:22:58.409521 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:22:58.409785 4702 external_mini_cluster.cc:1428] Setting key fc008d7c887488e5a71e9c0240f76fdf
I20260619 08:22:58.410986 24452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed7d3c6543eb4fa094c938caf47eb4e4: Bootstrap starting.
I20260619 08:22:58.411700 24452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed7d3c6543eb4fa094c938caf47eb4e4: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:58.412011 24452 log.cc:824] T 00000000000000000000000000000000 P ed7d3c6543eb4fa094c938caf47eb4e4: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:58.412683 24452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed7d3c6543eb4fa094c938caf47eb4e4: No bootstrap required, opened a new log
I20260619 08:22:58.415156 24452 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed7d3c6543eb4fa094c938caf47eb4e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7d3c6543eb4fa094c938caf47eb4e4" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 40751 } }
I20260619 08:22:58.415367 24452 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed7d3c6543eb4fa094c938caf47eb4e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:58.415417 24452 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed7d3c6543eb4fa094c938caf47eb4e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed7d3c6543eb4fa094c938caf47eb4e4, State: Initialized, Role: FOLLOWER
I20260619 08:22:58.415546 24452 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed7d3c6543eb4fa094c938caf47eb4e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7d3c6543eb4fa094c938caf47eb4e4" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 40751 } }
I20260619 08:22:58.415643 24452 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed7d3c6543eb4fa094c938caf47eb4e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:22:58.415681 24452 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed7d3c6543eb4fa094c938caf47eb4e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:22:58.415720 24452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed7d3c6543eb4fa094c938caf47eb4e4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:58.416216 24455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.410467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56902 (local address 127.4.151.190:40751)
0619 08:22:58.410727 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:58.410734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:58.410758 (+ 24us) server_negotiation.cc:407] Connection header received
0619 08:22:58.411205 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:58.411212 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:58.411272 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:58.411390 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:58.411693 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.412757 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.413888 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.414368 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.414488 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.415318 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:58.415321 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:58.415321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:22:58.415554 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:58.415589 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:22:58.415601 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:58.415686 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:58.415806 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 08:22:58.415997 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":113,"threads_started":1}
I20260619 08:22:58.416469 24452 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed7d3c6543eb4fa094c938caf47eb4e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7d3c6543eb4fa094c938caf47eb4e4" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 40751 } }
I20260619 08:22:58.416577 24452 leader_election.cc:302] T 00000000000000000000000000000000 P ed7d3c6543eb4fa094c938caf47eb4e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed7d3c6543eb4fa094c938caf47eb4e4; no voters:
I20260619 08:22:58.416738 24452 leader_election.cc:288] T 00000000000000000000000000000000 P ed7d3c6543eb4fa094c938caf47eb4e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:22:58.416774 24457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed7d3c6543eb4fa094c938caf47eb4e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:22:58.416999 24452 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed7d3c6543eb4fa094c938caf47eb4e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:22:58.417023 24457 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed7d3c6543eb4fa094c938caf47eb4e4 [term 1 LEADER]: Becoming Leader. State: Replica: ed7d3c6543eb4fa094c938caf47eb4e4, State: Running, Role: LEADER
I20260619 08:22:58.417097 24457 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed7d3c6543eb4fa094c938caf47eb4e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7d3c6543eb4fa094c938caf47eb4e4" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 40751 } }
I20260619 08:22:58.417433 24457 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed7d3c6543eb4fa094c938caf47eb4e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed7d3c6543eb4fa094c938caf47eb4e4. Latest consensus state: current_term: 1 leader_uuid: "ed7d3c6543eb4fa094c938caf47eb4e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7d3c6543eb4fa094c938caf47eb4e4" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 40751 } } }
I20260619 08:22:58.417456 24459 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed7d3c6543eb4fa094c938caf47eb4e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed7d3c6543eb4fa094c938caf47eb4e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7d3c6543eb4fa094c938caf47eb4e4" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 40751 } } }
I20260619 08:22:58.417520 24457 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed7d3c6543eb4fa094c938caf47eb4e4 [sys.catalog]: This master's current role is: LEADER
I20260619 08:22:58.417577 24459 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed7d3c6543eb4fa094c938caf47eb4e4 [sys.catalog]: This master's current role is: LEADER
I20260619 08:22:58.417757 24464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:22:58.418264 24464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:22:58.420004 24464 catalog_manager.cc:1347] Generated new cluster ID: c06a5f0b429849eeb09e96d560057bb8
I20260619 08:22:58.420060 24464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:22:58.433774 24464 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:22:58.434504 24464 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:22:58.441537 24464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed7d3c6543eb4fa094c938caf47eb4e4: Generated new TSK 0
I20260619 08:22:58.441807 24464 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:22:58.449561 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.129:0
--local_ip_for_outbound_sockets=127.4.151.129
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:40751
--builtin_ntp_servers=127.4.151.148:36605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 08:22:58.546945 24476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:58.547156 24476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:58.547179 24476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:58.548388 24476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:58.548432 24476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:58.548460 24476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.129
I20260619 08:22:58.549913 24476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:36605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.151.129
--webserver_port=0
--tserver_master_addrs=127.4.151.190:40751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.129
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:58.550174 24476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:58.550498 24476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:58.556656 24482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:58.556736 24484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:58.556934 24476 server_base.cc:1034] running on GCE node
W20260619 08:22:58.556663 24481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:58.557238 24476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:58.557542 24476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:58.558738 24476 hybrid_clock.cc:648] HybridClock initialized: now 1781857378558713 us; error 42 us; skew 500 ppm
I20260619 08:22:58.560083 24476 webserver.cc:460] Webserver started at http://127.4.151.129:38499/ using document root <none> and password file <none>
I20260619 08:22:58.560351 24476 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:58.560424 24476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:58.560582 24476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:22:58.561841 24476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-0/data/instance:
uuid: "e28f71c2099d46b7895e0295076da1c5"
format_stamp: "Formatted at 2026-06-19 08:22:58 on dist-test-slave-d3n8"
server_key: "f4f413e708fa973d36d7fe373ea544ed"
server_key_iv: "ab17f926a8f580c7dee67614e2d3d24d"
server_key_version: "encryptionkey@0"
I20260619 08:22:58.562250 24476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance:
uuid: "e28f71c2099d46b7895e0295076da1c5"
format_stamp: "Formatted at 2026-06-19 08:22:58 on dist-test-slave-d3n8"
server_key: "f4f413e708fa973d36d7fe373ea544ed"
server_key_iv: "ab17f926a8f580c7dee67614e2d3d24d"
server_key_version: "encryptionkey@0"
I20260619 08:22:58.563691 24476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:22:58.564548 24490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:58.564738 24476 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:22:58.564852 24476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-0/wal
uuid: "e28f71c2099d46b7895e0295076da1c5"
format_stamp: "Formatted at 2026-06-19 08:22:58 on dist-test-slave-d3n8"
server_key: "f4f413e708fa973d36d7fe373ea544ed"
server_key_iv: "ab17f926a8f580c7dee67614e2d3d24d"
server_key_version: "encryptionkey@0"
I20260619 08:22:58.564945 24476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:58.588281 24476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:58.588614 24476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:58.588778 24476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:58.589036 24476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:22:58.589457 24476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:22:58.589497 24476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:58.589520 24476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:22:58.589533 24476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:58.595733 24476 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.129:46193
I20260619 08:22:58.595793 24603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.129:46193 every 8 connection(s)
I20260619 08:22:58.596120 24476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-0/data/info.pb
I20260619 08:22:58.601920 24455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.596902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.129:46533 (local address 127.4.151.190:40751)
0619 08:22:58.596971 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:58.596973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:58.597206 (+ 233us) server_negotiation.cc:407] Connection header received
0619 08:22:58.597789 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:58.597790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:58.597805 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:58.597871 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:58.598565 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.599083 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.600366 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.600554 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.600624 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.601664 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:58.601665 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:58.601665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:22:58.601670 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:58.601686 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:22:58.601692 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:58.601747 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:58.601834 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 08:22:58.601859 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 08:22:58.602216 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.596888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40751 (local address 127.4.151.129:46533)
0619 08:22:58.597075 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:58.597117 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:58.597689 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:58.597930 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:58.597934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:58.597942 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:22:58.598445 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:58.598449 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.599445 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.599446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:58.600228 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:58.600230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.600670 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.600671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:58.600795 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.601496 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:22:58.601496 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:22:58.601524 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:22:58.601548 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:22:58.601759 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:22:58.601764 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:22:58.601766 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:22:58.601822 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 08:22:58.601907 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
I20260619 08:22:58.602787 24604 heartbeater.cc:347] Connected to a master server at 127.4.151.190:40751
I20260619 08:22:58.602880 24604 heartbeater.cc:464] Registering TS with master...
I20260619 08:22:58.603077 24604 heartbeater.cc:511] Master 127.4.151.190:40751 requested a full tablet report, sending...
I20260619 08:22:58.603762 24416 ts_manager.cc:194] Registered new tserver with Master: e28f71c2099d46b7895e0295076da1c5 (127.4.151.129:46193)
I20260619 08:22:58.604425 24416 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.151.129:46533
I20260619 08:22:58.605713 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 24476
I20260619 08:22:58.605826 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-0/wal/instance
I20260619 08:22:58.606040 4702 external_mini_cluster.cc:1428] Setting key dede39cd22d0bd171cfdd41d148f6ec7
I20260619 08:22:58.606962 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.130:0
--local_ip_for_outbound_sockets=127.4.151.130
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:40751
--builtin_ntp_servers=127.4.151.148:36605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 08:22:58.688894 24607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:58.689100 24607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:58.689122 24607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:58.690361 24607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:58.690410 24607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:58.690436 24607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.130
I20260619 08:22:58.691869 24607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:36605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.151.130
--webserver_port=0
--tserver_master_addrs=127.4.151.190:40751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.130
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:58.692116 24607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:58.692428 24607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:22:58.697573 24612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:58.697602 24615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:58.697576 24613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:58.697817 24607 server_base.cc:1034] running on GCE node
I20260619 08:22:58.698012 24607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:58.698201 24607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:58.699344 24607 hybrid_clock.cc:648] HybridClock initialized: now 1781857378699326 us; error 32 us; skew 500 ppm
I20260619 08:22:58.700806 24607 webserver.cc:460] Webserver started at http://127.4.151.130:46793/ using document root <none> and password file <none>
I20260619 08:22:58.701061 24607 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:58.701134 24607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:58.701296 24607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:22:58.702390 24607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-1/data/instance:
uuid: "e7d910aab24c4575bd12bd748b29d3f4"
format_stamp: "Formatted at 2026-06-19 08:22:58 on dist-test-slave-d3n8"
server_key: "ce73d045be9dbda2412eb54450b391ce"
server_key_iv: "36d399c7a2c0b71d26e7d9d90a92eb33"
server_key_version: "encryptionkey@0"
I20260619 08:22:58.702755 24607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance:
uuid: "e7d910aab24c4575bd12bd748b29d3f4"
format_stamp: "Formatted at 2026-06-19 08:22:58 on dist-test-slave-d3n8"
server_key: "ce73d045be9dbda2412eb54450b391ce"
server_key_iv: "36d399c7a2c0b71d26e7d9d90a92eb33"
server_key_version: "encryptionkey@0"
I20260619 08:22:58.704135 24607 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:22:58.705006 24621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:58.705231 24607 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:22:58.705338 24607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-1/wal
uuid: "e7d910aab24c4575bd12bd748b29d3f4"
format_stamp: "Formatted at 2026-06-19 08:22:58 on dist-test-slave-d3n8"
server_key: "ce73d045be9dbda2412eb54450b391ce"
server_key_iv: "36d399c7a2c0b71d26e7d9d90a92eb33"
server_key_version: "encryptionkey@0"
I20260619 08:22:58.705452 24607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:58.716042 24607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:58.716355 24607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:58.716506 24607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:58.716739 24607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:22:58.717113 24607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:22:58.717173 24607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:58.717236 24607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:22:58.717265 24607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:58.723387 24607 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.130:39029
I20260619 08:22:58.723429 24734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.130:39029 every 8 connection(s)
I20260619 08:22:58.723882 24607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-1/data/info.pb
I20260619 08:22:58.729598 24455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.724740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:38033 (local address 127.4.151.190:40751)
0619 08:22:58.724854 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:58.724857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:58.725048 (+ 191us) server_negotiation.cc:407] Connection header received
0619 08:22:58.725781 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:58.725782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:58.725795 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:58.725849 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:58.726560 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.727090 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.728115 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.728351 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.728434 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.729286 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:58.729287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:58.729288 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:22:58.729294 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:58.729310 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:22:58.729315 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:58.729358 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:58.729483 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 08:22:58.729512 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 08:22:58.729849 24737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.724637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40751 (local address 127.4.151.130:38033)
0619 08:22:58.724922 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:58.724958 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:58.725647 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:58.725925 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:58.725928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:58.725936 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:22:58.726463 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:58.726468 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.727201 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.727202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:58.728016 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:58.728018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.728443 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.728444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:58.728581 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.729131 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:22:58.729132 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:22:58.729160 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:22:58.729184 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:22:58.729360 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:22:58.729366 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:22:58.729392 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 08:22:58.729485 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 08:22:58.729592 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":109,"threads_started":1}
I20260619 08:22:58.730415 24735 heartbeater.cc:347] Connected to a master server at 127.4.151.190:40751
I20260619 08:22:58.730571 24735 heartbeater.cc:464] Registering TS with master...
I20260619 08:22:58.730773 24735 heartbeater.cc:511] Master 127.4.151.190:40751 requested a full tablet report, sending...
I20260619 08:22:58.731246 24416 ts_manager.cc:194] Registered new tserver with Master: e7d910aab24c4575bd12bd748b29d3f4 (127.4.151.130:39029)
I20260619 08:22:58.731637 24416 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.151.130:38033
I20260619 08:22:58.732010 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 24607
I20260619 08:22:58.732125 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-1/wal/instance
I20260619 08:22:58.732338 4702 external_mini_cluster.cc:1428] Setting key e459fa6f94b797886b049f6e7a99bbe4
I20260619 08:22:58.733361 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.151.131:0
--local_ip_for_outbound_sockets=127.4.151.131
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:40751
--builtin_ntp_servers=127.4.151.148:36605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 08:22:58.823514 24738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:22:58.823801 24738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:22:58.823844 24738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:22:58.825124 24738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:22:58.825232 24738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:22:58.825282 24738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.151.131
I20260619 08:22:58.827114 24738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:36605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.151.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.151.131
--webserver_port=0
--tserver_master_addrs=127.4.151.190:40751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.151.131
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:22:58.827436 24738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:22:58.827764 24738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:22:58.833520 24738 server_base.cc:1034] running on GCE node
W20260619 08:22:58.833523 24746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:58.833567 24744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:22:58.833523 24743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:22:58.834020 24738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:22:58.834302 24738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:22:58.835471 24738 hybrid_clock.cc:648] HybridClock initialized: now 1781857378835463 us; error 36 us; skew 500 ppm
I20260619 08:22:58.836808 24738 webserver.cc:460] Webserver started at http://127.4.151.131:46053/ using document root <none> and password file <none>
I20260619 08:22:58.837071 24738 fs_manager.cc:359] Metadata directory not provided
I20260619 08:22:58.837145 24738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:22:58.837298 24738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:22:58.838387 24738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-2/data/instance:
uuid: "2edf4efb331c4d50b2a5e1fb13d20e74"
format_stamp: "Formatted at 2026-06-19 08:22:58 on dist-test-slave-d3n8"
server_key: "dc4dbe5a0b6be6f5cd808f10499a066c"
server_key_iv: "9369eee3c512ff6452e3a76de2415787"
server_key_version: "encryptionkey@0"
I20260619 08:22:58.838757 24738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance:
uuid: "2edf4efb331c4d50b2a5e1fb13d20e74"
format_stamp: "Formatted at 2026-06-19 08:22:58 on dist-test-slave-d3n8"
server_key: "dc4dbe5a0b6be6f5cd808f10499a066c"
server_key_iv: "9369eee3c512ff6452e3a76de2415787"
server_key_version: "encryptionkey@0"
I20260619 08:22:58.840029 24738 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:22:58.840821 24752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:58.840993 24738 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:22:58.841096 24738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-2/wal
uuid: "2edf4efb331c4d50b2a5e1fb13d20e74"
format_stamp: "Formatted at 2026-06-19 08:22:58 on dist-test-slave-d3n8"
server_key: "dc4dbe5a0b6be6f5cd808f10499a066c"
server_key_iv: "9369eee3c512ff6452e3a76de2415787"
server_key_version: "encryptionkey@0"
I20260619 08:22:58.841174 24738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:22:58.867856 24738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:22:58.868232 24738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:22:58.868415 24738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:22:58.868669 24738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:22:58.869056 24738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:22:58.869118 24738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:58.869154 24738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:22:58.869181 24738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:22:58.874976 24738 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.131:39189
I20260619 08:22:58.875020 24865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.131:39189 every 8 connection(s)
I20260619 08:22:58.875347 24738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-2/data/info.pb
I20260619 08:22:58.879101 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 24738
I20260619 08:22:58.879215 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-2/wal/instance
I20260619 08:22:58.879487 4702 external_mini_cluster.cc:1428] Setting key f66794702141ccdfe7aaa53a63b02c46
I20260619 08:22:58.882042 24455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.876124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.131:45683 (local address 127.4.151.190:40751)
0619 08:22:58.876291 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:58.876294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:58.876501 (+ 207us) server_negotiation.cc:407] Connection header received
0619 08:22:58.877035 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:58.877036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:58.877047 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:58.877109 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:58.877947 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.878562 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.880168 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.880495 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.880581 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.881689 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:58.881690 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:58.881691 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:22:58.881696 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:58.881715 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:22:58.881720 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:58.881782 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:58.881892 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 08:22:58.881925 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260619 08:22:58.882298 24868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.876119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:40751 (local address 127.4.151.131:45683)
0619 08:22:58.876396 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:58.876432 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:58.876928 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:58.877201 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:58.877204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:58.877212 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:22:58.877835 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:58.877840 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.878683 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.878684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:58.880061 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:58.880064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.880616 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.880617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:58.880822 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.881532 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:22:58.881532 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:22:58.881570 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:22:58.881605 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:22:58.881790 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:22:58.881795 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:22:58.881797 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:22:58.881891 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 08:22:58.882014 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":98,"threads_started":1}
I20260619 08:22:58.882834 24866 heartbeater.cc:347] Connected to a master server at 127.4.151.190:40751
I20260619 08:22:58.882961 24866 heartbeater.cc:464] Registering TS with master...
I20260619 08:22:58.883189 24866 heartbeater.cc:511] Master 127.4.151.190:40751 requested a full tablet report, sending...
I20260619 08:22:58.883644 24416 ts_manager.cc:194] Registered new tserver with Master: 2edf4efb331c4d50b2a5e1fb13d20e74 (127.4.151.131:39189)
I20260619 08:22:58.884156 24416 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.151.131:45683
I20260619 08:22:58.892012 4702 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:22:58.897589 24455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.893535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56916 (local address 127.4.151.190:40751)
0619 08:22:58.894655 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:58.894657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:58.894665 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:22:58.894711 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:58.894712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:58.894723 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:58.894778 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:22:58.894973 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.895484 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.896287 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.896496 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.896577 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.897302 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:22:58.897303 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:22:58.897304 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:22:58.897308 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:22:58.897327 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:22:58.897331 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:22:58.897408 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:58.897470 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 08:22:58.897494 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1095}
I20260619 08:22:58.899221 24416 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56916:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:22:58.900836 24416 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:22:58.907531 24878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.903567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52186 (local address 127.4.151.130:39029)
0619 08:22:58.903978 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:58.903984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:58.903994 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:22:58.904032 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:58.904034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:58.904048 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:58.904113 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:58.904313 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.905420 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.906614 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.907117 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.907223 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.907319 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:58.907405 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 08:22:58.907444 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":68,"threads_started":1}
I20260619 08:22:58.907723 24876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.903370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48084 (local address 127.4.151.129:46193)
0619 08:22:58.903792 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:58.903797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:58.903811 (+ 14us) server_negotiation.cc:407] Connection header received
0619 08:22:58.903933 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:58.903940 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:58.903959 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:58.904039 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:58.904260 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.904983 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.906436 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.907192 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.907264 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.907362 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:58.907598 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 08:22:58.907639 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":84,"threads_started":1}
I20260619 08:22:58.907851 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.903613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33600 (local address 127.4.151.131:39189)
0619 08:22:58.903810 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:58.903815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:58.903826 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:22:58.903869 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:58.903871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:58.903886 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:58.904188 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:58.904248 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.905432 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.906587 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.907267 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.907603 (+ 336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.907690 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:58.907714 (+ 24us) server_negotiation.cc:299] Negotiation successful
0619 08:22:58.907759 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":86,"threads_started":1}
I20260619 08:22:58.907521 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.903476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.130:39029 (local address 127.0.0.1:52186)
0619 08:22:58.903794 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:58.903805 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:58.903858 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:58.904189 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:58.904190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:58.904192 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:58.904228 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:58.904229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.905496 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.905497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:58.906532 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:58.906534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.907228 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.907229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:58.907303 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.907317 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 08:22:58.907391 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 08:22:58.907445 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":67,"threads_started":1}
I20260619 08:22:58.907960 24874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.903323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:39189 (local address 127.0.0.1:33600)
0619 08:22:58.903531 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:58.903540 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:58.903582 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:58.904024 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:58.904027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:58.904030 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:58.904110 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:58.904112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.905532 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.905533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:58.906489 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:58.906491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.907360 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.907361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:58.907433 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.907437 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:22:58.907470 (+ 33us) client_negotiation.cc:240] Negotiation successful
0619 08:22:58.907512 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":9,"thread_start_us":70,"threads_started":1}
I20260619 08:22:58.908061 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.903260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:46193 (local address 127.0.0.1:48084)
0619 08:22:58.903466 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:58.903510 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:58.903582 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:58.904118 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:58.904119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:58.904121 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:58.904178 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:58.904180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.905090 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.905092 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:58.906345 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:58.906347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.907391 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.907391 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:58.907438 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.907441 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:22:58.907486 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 08:22:58.907517 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
I20260619 08:22:58.908712 24800 tablet_service.cc:1467] Processing CreateTablet for tablet b144f5a504344d9ea2f83c1c1cde08e2 (DEFAULT_TABLE table=test-table [id=952e90e6d36c4c8c859ef4748cfb1f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:22:58.908747 24538 tablet_service.cc:1467] Processing CreateTablet for tablet b144f5a504344d9ea2f83c1c1cde08e2 (DEFAULT_TABLE table=test-table [id=952e90e6d36c4c8c859ef4748cfb1f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:22:58.908747 24669 tablet_service.cc:1467] Processing CreateTablet for tablet b144f5a504344d9ea2f83c1c1cde08e2 (DEFAULT_TABLE table=test-table [id=952e90e6d36c4c8c859ef4748cfb1f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:22:58.908994 24800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b144f5a504344d9ea2f83c1c1cde08e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:58.909075 24669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b144f5a504344d9ea2f83c1c1cde08e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:58.909075 24538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b144f5a504344d9ea2f83c1c1cde08e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:22:58.911831 24880 tablet_bootstrap.cc:492] T b144f5a504344d9ea2f83c1c1cde08e2 P 2edf4efb331c4d50b2a5e1fb13d20e74: Bootstrap starting.
I20260619 08:22:58.912328 24882 tablet_bootstrap.cc:492] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4: Bootstrap starting.
I20260619 08:22:58.912529 24880 tablet_bootstrap.cc:654] T b144f5a504344d9ea2f83c1c1cde08e2 P 2edf4efb331c4d50b2a5e1fb13d20e74: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:58.912868 24880 log.cc:824] T b144f5a504344d9ea2f83c1c1cde08e2 P 2edf4efb331c4d50b2a5e1fb13d20e74: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:58.912951 24882 tablet_bootstrap.cc:654] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:58.913229 24882 log.cc:824] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:58.913592 24881 tablet_bootstrap.cc:492] T b144f5a504344d9ea2f83c1c1cde08e2 P e28f71c2099d46b7895e0295076da1c5: Bootstrap starting.
I20260619 08:22:58.913646 24880 tablet_bootstrap.cc:492] T b144f5a504344d9ea2f83c1c1cde08e2 P 2edf4efb331c4d50b2a5e1fb13d20e74: No bootstrap required, opened a new log
I20260619 08:22:58.913699 24880 ts_tablet_manager.cc:1397] T b144f5a504344d9ea2f83c1c1cde08e2 P 2edf4efb331c4d50b2a5e1fb13d20e74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:22:58.913934 24882 tablet_bootstrap.cc:492] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4: No bootstrap required, opened a new log
I20260619 08:22:58.914016 24882 ts_tablet_manager.cc:1397] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 08:22:58.914256 24881 tablet_bootstrap.cc:654] T b144f5a504344d9ea2f83c1c1cde08e2 P e28f71c2099d46b7895e0295076da1c5: Neither blocks nor log segments found. Creating new log.
I20260619 08:22:58.914588 24881 log.cc:824] T b144f5a504344d9ea2f83c1c1cde08e2 P e28f71c2099d46b7895e0295076da1c5: Log is configured to *not* fsync() on all Append() calls
I20260619 08:22:58.915393 24881 tablet_bootstrap.cc:492] T b144f5a504344d9ea2f83c1c1cde08e2 P e28f71c2099d46b7895e0295076da1c5: No bootstrap required, opened a new log
I20260619 08:22:58.915463 24881 ts_tablet_manager.cc:1397] T b144f5a504344d9ea2f83c1c1cde08e2 P e28f71c2099d46b7895e0295076da1c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:22:58.916267 24882 raft_consensus.cc:348] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28f71c2099d46b7895e0295076da1c5" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 46193 } } peers { permanent_uuid: "2edf4efb331c4d50b2a5e1fb13d20e74" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39189 } } peers { permanent_uuid: "e7d910aab24c4575bd12bd748b29d3f4" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39029 } }
I20260619 08:22:58.916424 24882 raft_consensus.cc:374] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:58.916463 24882 raft_consensus.cc:729] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7d910aab24c4575bd12bd748b29d3f4, State: Initialized, Role: FOLLOWER
I20260619 08:22:58.916594 24882 consensus_queue.cc:260] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28f71c2099d46b7895e0295076da1c5" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 46193 } } peers { permanent_uuid: "2edf4efb331c4d50b2a5e1fb13d20e74" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39189 } } peers { permanent_uuid: "e7d910aab24c4575bd12bd748b29d3f4" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39029 } }
I20260619 08:22:58.916571 24880 raft_consensus.cc:348] T b144f5a504344d9ea2f83c1c1cde08e2 P 2edf4efb331c4d50b2a5e1fb13d20e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28f71c2099d46b7895e0295076da1c5" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 46193 } } peers { permanent_uuid: "2edf4efb331c4d50b2a5e1fb13d20e74" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39189 } } peers { permanent_uuid: "e7d910aab24c4575bd12bd748b29d3f4" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39029 } }
I20260619 08:22:58.916787 24880 raft_consensus.cc:374] T b144f5a504344d9ea2f83c1c1cde08e2 P 2edf4efb331c4d50b2a5e1fb13d20e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:58.916836 24880 raft_consensus.cc:729] T b144f5a504344d9ea2f83c1c1cde08e2 P 2edf4efb331c4d50b2a5e1fb13d20e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edf4efb331c4d50b2a5e1fb13d20e74, State: Initialized, Role: FOLLOWER
I20260619 08:22:58.916869 24882 ts_tablet_manager.cc:1428] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 08:22:58.916998 24735 heartbeater.cc:503] Master 127.4.151.190:40751 was elected leader, sending a full tablet report...
I20260619 08:22:58.916973 24880 consensus_queue.cc:260] T b144f5a504344d9ea2f83c1c1cde08e2 P 2edf4efb331c4d50b2a5e1fb13d20e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28f71c2099d46b7895e0295076da1c5" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 46193 } } peers { permanent_uuid: "2edf4efb331c4d50b2a5e1fb13d20e74" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39189 } } peers { permanent_uuid: "e7d910aab24c4575bd12bd748b29d3f4" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39029 } }
I20260619 08:22:58.917224 24880 ts_tablet_manager.cc:1428] T b144f5a504344d9ea2f83c1c1cde08e2 P 2edf4efb331c4d50b2a5e1fb13d20e74: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260619 08:22:58.917349 24866 heartbeater.cc:503] Master 127.4.151.190:40751 was elected leader, sending a full tablet report...
I20260619 08:22:58.918535 24881 raft_consensus.cc:348] T b144f5a504344d9ea2f83c1c1cde08e2 P e28f71c2099d46b7895e0295076da1c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28f71c2099d46b7895e0295076da1c5" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 46193 } } peers { permanent_uuid: "2edf4efb331c4d50b2a5e1fb13d20e74" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39189 } } peers { permanent_uuid: "e7d910aab24c4575bd12bd748b29d3f4" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39029 } }
I20260619 08:22:58.918702 24881 raft_consensus.cc:374] T b144f5a504344d9ea2f83c1c1cde08e2 P e28f71c2099d46b7895e0295076da1c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:22:58.918738 24881 raft_consensus.cc:729] T b144f5a504344d9ea2f83c1c1cde08e2 P e28f71c2099d46b7895e0295076da1c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e28f71c2099d46b7895e0295076da1c5, State: Initialized, Role: FOLLOWER
I20260619 08:22:58.918850 24881 consensus_queue.cc:260] T b144f5a504344d9ea2f83c1c1cde08e2 P e28f71c2099d46b7895e0295076da1c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28f71c2099d46b7895e0295076da1c5" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 46193 } } peers { permanent_uuid: "2edf4efb331c4d50b2a5e1fb13d20e74" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39189 } } peers { permanent_uuid: "e7d910aab24c4575bd12bd748b29d3f4" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39029 } }
I20260619 08:22:58.919062 24881 ts_tablet_manager.cc:1428] T b144f5a504344d9ea2f83c1c1cde08e2 P e28f71c2099d46b7895e0295076da1c5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 08:22:58.919147 24604 heartbeater.cc:503] Master 127.4.151.190:40751 was elected leader, sending a full tablet report...
I20260619 08:22:58.945950 24886 raft_consensus.cc:482] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:22:58.946065 24886 raft_consensus.cc:504] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28f71c2099d46b7895e0295076da1c5" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 46193 } } peers { permanent_uuid: "2edf4efb331c4d50b2a5e1fb13d20e74" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39189 } } peers { permanent_uuid: "e7d910aab24c4575bd12bd748b29d3f4" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39029 } }
I20260619 08:22:58.946442 24886 leader_election.cc:288] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e28f71c2099d46b7895e0295076da1c5 (127.4.151.129:46193), 2edf4efb331c4d50b2a5e1fb13d20e74 (127.4.151.131:39189)
I20260619 08:22:58.949682 24876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.946726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:40259 (local address 127.4.151.129:46193)
0619 08:22:58.946767 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:58.946769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:58.946778 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:22:58.946809 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:58.946810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:58.946821 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:58.946899 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:58.947097 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.947669 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.948853 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.949295 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.949351 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.949358 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:58.949556 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 08:22:58.949588 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:22:58.949694 24737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.946625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:46193 (local address 127.4.151.130:40259)
0619 08:22:58.946666 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:58.946694 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:58.946738 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:58.946927 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:58.946928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:58.946929 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:58.947010 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:58.947012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.947746 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.947747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:58.948779 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:58.948782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.949400 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.949401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:58.949476 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.949480 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:22:58.949543 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 08:22:58.949590 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":1}
I20260619 08:22:58.949958 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.946706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.151.130:46167 (local address 127.4.151.131:39189)
0619 08:22:58.946751 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:58.946754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:58.946917 (+ 163us) server_negotiation.cc:407] Connection header received
0619 08:22:58.946956 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:58.946957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:58.946966 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:58.947029 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:22:58.947304 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.947784 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.949051 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.949560 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.949623 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.949630 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:58.949842 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 08:22:58.949889 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:22:58.950068 24558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b144f5a504344d9ea2f83c1c1cde08e2" candidate_uuid: "e7d910aab24c4575bd12bd748b29d3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e28f71c2099d46b7895e0295076da1c5" is_pre_election: true
I20260619 08:22:58.950259 24889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.946625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.131:39189 (local address 127.4.151.130:46167)
0619 08:22:58.946838 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:58.946849 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:22:58.946908 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:22:58.947089 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:22:58.947090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:22:58.947091 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:22:58.947228 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:58.947230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.947890 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.947891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:58.948967 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:22:58.948970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.949647 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.949648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:22:58.949737 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.949741 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:22:58.950099 (+ 358us) client_negotiation.cc:240] Negotiation successful
0619 08:22:58.950141 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":47,"thread_start_us":105,"threads_started":1}
I20260619 08:22:58.950265 24558 raft_consensus.cc:2454] T b144f5a504344d9ea2f83c1c1cde08e2 P e28f71c2099d46b7895e0295076da1c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7d910aab24c4575bd12bd748b29d3f4 in term 0.
I20260619 08:22:58.950546 24624 leader_election.cc:302] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e28f71c2099d46b7895e0295076da1c5, e7d910aab24c4575bd12bd748b29d3f4; no voters:
I20260619 08:22:58.950592 24820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b144f5a504344d9ea2f83c1c1cde08e2" candidate_uuid: "e7d910aab24c4575bd12bd748b29d3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edf4efb331c4d50b2a5e1fb13d20e74" is_pre_election: true
I20260619 08:22:58.950672 24886 raft_consensus.cc:2781] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:22:58.950721 24886 raft_consensus.cc:482] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:22:58.950753 24886 raft_consensus.cc:3037] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:58.950766 24820 raft_consensus.cc:2454] T b144f5a504344d9ea2f83c1c1cde08e2 P 2edf4efb331c4d50b2a5e1fb13d20e74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7d910aab24c4575bd12bd748b29d3f4 in term 0.
I20260619 08:22:58.951476 24886 raft_consensus.cc:504] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28f71c2099d46b7895e0295076da1c5" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 46193 } } peers { permanent_uuid: "2edf4efb331c4d50b2a5e1fb13d20e74" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39189 } } peers { permanent_uuid: "e7d910aab24c4575bd12bd748b29d3f4" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39029 } }
I20260619 08:22:58.951630 24886 leader_election.cc:288] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4 [CANDIDATE]: Term 1 election: Requested vote from peers e28f71c2099d46b7895e0295076da1c5 (127.4.151.129:46193), 2edf4efb331c4d50b2a5e1fb13d20e74 (127.4.151.131:39189)
I20260619 08:22:58.951784 24820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b144f5a504344d9ea2f83c1c1cde08e2" candidate_uuid: "e7d910aab24c4575bd12bd748b29d3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edf4efb331c4d50b2a5e1fb13d20e74"
I20260619 08:22:58.951798 24558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b144f5a504344d9ea2f83c1c1cde08e2" candidate_uuid: "e7d910aab24c4575bd12bd748b29d3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e28f71c2099d46b7895e0295076da1c5"
I20260619 08:22:58.951861 24820 raft_consensus.cc:3037] T b144f5a504344d9ea2f83c1c1cde08e2 P 2edf4efb331c4d50b2a5e1fb13d20e74 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:58.951875 24558 raft_consensus.cc:3037] T b144f5a504344d9ea2f83c1c1cde08e2 P e28f71c2099d46b7895e0295076da1c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:22:58.952577 24820 raft_consensus.cc:2454] T b144f5a504344d9ea2f83c1c1cde08e2 P 2edf4efb331c4d50b2a5e1fb13d20e74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7d910aab24c4575bd12bd748b29d3f4 in term 1.
I20260619 08:22:58.952751 24623 leader_election.cc:302] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edf4efb331c4d50b2a5e1fb13d20e74, e7d910aab24c4575bd12bd748b29d3f4; no voters:
I20260619 08:22:58.952770 24558 raft_consensus.cc:2454] T b144f5a504344d9ea2f83c1c1cde08e2 P e28f71c2099d46b7895e0295076da1c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7d910aab24c4575bd12bd748b29d3f4 in term 1.
I20260619 08:22:58.952853 24886 raft_consensus.cc:2781] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:22:58.953008 24886 raft_consensus.cc:686] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4 [term 1 LEADER]: Becoming Leader. State: Replica: e7d910aab24c4575bd12bd748b29d3f4, State: Running, Role: LEADER
I20260619 08:22:58.953095 24886 consensus_queue.cc:237] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28f71c2099d46b7895e0295076da1c5" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 46193 } } peers { permanent_uuid: "2edf4efb331c4d50b2a5e1fb13d20e74" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39189 } } peers { permanent_uuid: "e7d910aab24c4575bd12bd748b29d3f4" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39029 } }
I20260619 08:22:58.953938 24416 catalog_manager.cc:5495] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7d910aab24c4575bd12bd748b29d3f4 (127.4.151.130). New cstate: current_term: 1 leader_uuid: "e7d910aab24c4575bd12bd748b29d3f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28f71c2099d46b7895e0295076da1c5" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edf4efb331c4d50b2a5e1fb13d20e74" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7d910aab24c4575bd12bd748b29d3f4" member_type: VOTER last_known_addr { host: "127.4.151.130" port: 39029 } health_report { overall_health: HEALTHY } } }
I20260619 08:22:58.965225 24878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.962967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52198 (local address 127.4.151.130:39029)
0619 08:22:58.963068 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 08:22:58.963070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:22:58.963079 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:22:58.963111 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:22:58.963111 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:22:58.963130 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:22:58.963199 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:22:58.963319 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.963786 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.964503 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:22:58.964691 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:22:58.964752 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:22:58.964873 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:22:58.964974 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:22:58.965034 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 08:22:58.965066 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 08:22:58.965083 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 08:22:58.967289 24558 raft_consensus.cc:1261] T b144f5a504344d9ea2f83c1c1cde08e2 P e28f71c2099d46b7895e0295076da1c5 [term 1 FOLLOWER]: Refusing update from remote peer e7d910aab24c4575bd12bd748b29d3f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:22:58.967289 24820 raft_consensus.cc:1261] T b144f5a504344d9ea2f83c1c1cde08e2 P 2edf4efb331c4d50b2a5e1fb13d20e74 [term 1 FOLLOWER]: Refusing update from remote peer e7d910aab24c4575bd12bd748b29d3f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:22:58.967667 24886 consensus_queue.cc:1035] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e28f71c2099d46b7895e0295076da1c5" member_type: VOTER last_known_addr { host: "127.4.151.129" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:22:58.967820 24890 consensus_queue.cc:1035] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edf4efb331c4d50b2a5e1fb13d20e74" member_type: VOTER last_known_addr { host: "127.4.151.131" port: 39189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:22:58.971244 24895 mvcc.cc:205] Tried to move back new op lower bound from 7298487824246902784 to 7298487824192450560. Current Snapshot: MvccSnapshot[applied={T|T < 7298487824246902784 or (T in {7298487824246902784})}]
I20260619 08:22:58.973599 24415 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:22:58.973703 24415 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:22:58.974781 24415 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed7d3c6543eb4fa094c938caf47eb4e4: Sending DeleteTablet for 3 replicas of tablet b144f5a504344d9ea2f83c1c1cde08e2
I20260619 08:22:58.975117 24669 tablet_service.cc:1514] Processing DeleteTablet for tablet b144f5a504344d9ea2f83c1c1cde08e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:22:58 UTC) from {username='slave'} at 127.0.0.1:52186
I20260619 08:22:58.975117 24538 tablet_service.cc:1514] Processing DeleteTablet for tablet b144f5a504344d9ea2f83c1c1cde08e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:22:58 UTC) from {username='slave'} at 127.0.0.1:48084
I20260619 08:22:58.975162 24800 tablet_service.cc:1514] Processing DeleteTablet for tablet b144f5a504344d9ea2f83c1c1cde08e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:22:58 UTC) from {username='slave'} at 127.0.0.1:33600
I20260619 08:22:58.975567 24904 tablet_replica.cc:330] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4: stopping tablet replica
I20260619 08:22:58.975672 24904 raft_consensus.cc:2229] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:22:58.975770 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 24476
I20260619 08:22:58.975797 24904 raft_consensus.cc:2258] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:22:58.978207 24904 ts_tablet_manager.cc:1905] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:22:58.979930 24904 ts_tablet_manager.cc:1918] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:22:58.980037 24904 log.cc:1196] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-1/wal/wals/b144f5a504344d9ea2f83c1c1cde08e2
I20260619 08:22:58.980309 24904 ts_tablet_manager.cc:1939] T b144f5a504344d9ea2f83c1c1cde08e2 P e7d910aab24c4575bd12bd748b29d3f4: Deleting consensus metadata
I20260619 08:22:58.980790 24402 catalog_manager.cc:4841] TS e7d910aab24c4575bd12bd748b29d3f4 (127.4.151.130:39029): tablet b144f5a504344d9ea2f83c1c1cde08e2 (table test-table [id=952e90e6d36c4c8c859ef4748cfb1f6a]) successfully deleted
W20260619 08:22:58.982825 24402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.151.129:46193 (error 108)
I20260619 08:22:58.983080 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 24607
I20260619 08:22:58.985044 24905 tablet_replica.cc:330] T b144f5a504344d9ea2f83c1c1cde08e2 P 2edf4efb331c4d50b2a5e1fb13d20e74: stopping tablet replica
I20260619 08:22:58.985177 24905 raft_consensus.cc:2229] T b144f5a504344d9ea2f83c1c1cde08e2 P 2edf4efb331c4d50b2a5e1fb13d20e74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:22:58.985198 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:22:58.985050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.129:46193 (local address 127.0.0.1:48100)
0619 08:22:58.985112 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 08:22:58.985162 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.129:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 08:22:58.985307 24905 raft_consensus.cc:2258] T b144f5a504344d9ea2f83c1c1cde08e2 P 2edf4efb331c4d50b2a5e1fb13d20e74 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 08:22:58.985314 24402 catalog_manager.cc:4584] TS e28f71c2099d46b7895e0295076da1c5 (127.4.151.129:46193): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b144f5a504344d9ea2f83c1c1cde08e2: Network error: Client connection negotiation failed: client connection to 127.4.151.129:46193: connect: Connection refused (error 111)
I20260619 08:22:58.985863 24905 ts_tablet_manager.cc:1905] T b144f5a504344d9ea2f83c1c1cde08e2 P 2edf4efb331c4d50b2a5e1fb13d20e74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:22:58.987877 24905 ts_tablet_manager.cc:1918] T b144f5a504344d9ea2f83c1c1cde08e2 P 2edf4efb331c4d50b2a5e1fb13d20e74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:22:58.987954 24905 log.cc:1196] T b144f5a504344d9ea2f83c1c1cde08e2 P 2edf4efb331c4d50b2a5e1fb13d20e74: Deleting WAL directory at /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781857130600961-4702-0/minicluster-data/ts-2/wal/wals/b144f5a504344d9ea2f83c1c1cde08e2
I20260619 08:22:58.988199 24905 ts_tablet_manager.cc:1939] T b144f5a504344d9ea2f83c1c1cde08e2 P 2edf4efb331c4d50b2a5e1fb13d20e74: Deleting consensus metadata
I20260619 08:22:58.988715 24401 catalog_manager.cc:4841] TS 2edf4efb331c4d50b2a5e1fb13d20e74 (127.4.151.131:39189): tablet b144f5a504344d9ea2f83c1c1cde08e2 (table test-table [id=952e90e6d36c4c8c859ef4748cfb1f6a]) successfully deleted
I20260619 08:22:58.989106 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 24738
I20260619 08:22:58.994488 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 24385
2026-06-19T08:22:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (736 ms)
[----------] 6 tests from EncryptionPolicyTest (8851 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:22:59 dist-test-slave-d3n8 krb5kdc[24909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:22:59 dist-test-slave-d3n8 krb5kdc[24909](info): set up 2 sockets
Jun 19 08:22:59 dist-test-slave-d3n8 krb5kdc[24909](info): commencing operation
krb5kdc: starting...
W20260619 08:23:00.042011 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:23:00 dist-test-slave-d3n8 krb5kdc[24909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:23:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:23:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 08:23:00.169183 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42613
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:23:00.259367 24922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:23:00.259580 24922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:23:00.259603 24922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:23:00.260795 24922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:23:00.260841 24922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:23:00.260855 24922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:23:00.260869 24922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:23:00.260880 24922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:23:00.262423 24922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42613
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:23:00.262673 24922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:23:00.262944 24922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:23:00.268302 24930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:00.268309 24927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:00.268311 24928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:00.268674 24922 server_base.cc:1034] running on GCE node
I20260619 08:23:00.268910 24922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:23:00.269235 24922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:23:00.270430 24922 hybrid_clock.cc:648] HybridClock initialized: now 1781857380270429 us; error 43 us; skew 500 ppm
Jun 19 08:23:00 dist-test-slave-d3n8 krb5kdc[24909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:23:00.273229 24922 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 08:23:00.273816 24922 webserver.cc:460] Webserver started at http://127.0.0.1:35297/ using document root <none> and password file <none>
I20260619 08:23:00.274101 24922 fs_manager.cc:359] Metadata directory not provided
I20260619 08:23:00.274175 24922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:23:00.274338 24922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:23:00.275429 24922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "6fb785ee6daa43468ea4795aff8a6773"
format_stamp: "Formatted at 2026-06-19 08:23:00 on dist-test-slave-d3n8"
server_key: "0138f261fcb5b6627c3c5c964a9cb9df"
server_key_iv: "f57de8237ce6c0e9ce78bb5e7008c4f2"
server_key_version: "encryptionkey@0"
I20260619 08:23:00.275808 24922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "6fb785ee6daa43468ea4795aff8a6773"
format_stamp: "Formatted at 2026-06-19 08:23:00 on dist-test-slave-d3n8"
server_key: "0138f261fcb5b6627c3c5c964a9cb9df"
server_key_iv: "f57de8237ce6c0e9ce78bb5e7008c4f2"
server_key_version: "encryptionkey@0"
I20260619 08:23:00.277204 24922 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:23:00.278103 24937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:23:00.278501 24922 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:23:00.278618 24922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "6fb785ee6daa43468ea4795aff8a6773"
format_stamp: "Formatted at 2026-06-19 08:23:00 on dist-test-slave-d3n8"
server_key: "0138f261fcb5b6627c3c5c964a9cb9df"
server_key_iv: "f57de8237ce6c0e9ce78bb5e7008c4f2"
server_key_version: "encryptionkey@0"
I20260619 08:23:00.278719 24922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:23:00.322427 24922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:23:00.324682 24922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:23:00.324877 24922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:23:00.329032 24922 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42613
I20260619 08:23:00.329069 24989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42613 every 8 connection(s)
I20260619 08:23:00.329494 24922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:23:00.330504 24990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:23:00.332769 24990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb785ee6daa43468ea4795aff8a6773: Bootstrap starting.
I20260619 08:23:00.333310 24990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fb785ee6daa43468ea4795aff8a6773: Neither blocks nor log segments found. Creating new log.
I20260619 08:23:00.333626 24990 log.cc:824] T 00000000000000000000000000000000 P 6fb785ee6daa43468ea4795aff8a6773: Log is configured to *not* fsync() on all Append() calls
I20260619 08:23:00.334417 24990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb785ee6daa43468ea4795aff8a6773: No bootstrap required, opened a new log
I20260619 08:23:00.335915 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 24922
I20260619 08:23:00.336045 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:23:00.336251 4702 external_mini_cluster.cc:1428] Setting key 2b12d84bd69f9c48561676bc60b693f5
I20260619 08:23:00.336329 24990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fb785ee6daa43468ea4795aff8a6773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb785ee6daa43468ea4795aff8a6773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42613 } }
I20260619 08:23:00.336521 24990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fb785ee6daa43468ea4795aff8a6773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:23:00.336570 24990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fb785ee6daa43468ea4795aff8a6773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb785ee6daa43468ea4795aff8a6773, State: Initialized, Role: FOLLOWER
I20260619 08:23:00.336730 24990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fb785ee6daa43468ea4795aff8a6773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb785ee6daa43468ea4795aff8a6773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42613 } }
I20260619 08:23:00.336819 24990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fb785ee6daa43468ea4795aff8a6773 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:23:00.336856 24990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fb785ee6daa43468ea4795aff8a6773 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:23:00.336905 24990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fb785ee6daa43468ea4795aff8a6773 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:23:00.337549 24990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fb785ee6daa43468ea4795aff8a6773 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb785ee6daa43468ea4795aff8a6773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42613 } }
I20260619 08:23:00.337667 24990 leader_election.cc:302] T 00000000000000000000000000000000 P 6fb785ee6daa43468ea4795aff8a6773 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fb785ee6daa43468ea4795aff8a6773; no voters:
I20260619 08:23:00.337852 24990 leader_election.cc:288] T 00000000000000000000000000000000 P 6fb785ee6daa43468ea4795aff8a6773 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:23:00.337873 24995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fb785ee6daa43468ea4795aff8a6773 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:23:00.337996 24995 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fb785ee6daa43468ea4795aff8a6773 [term 1 LEADER]: Becoming Leader. State: Replica: 6fb785ee6daa43468ea4795aff8a6773, State: Running, Role: LEADER
I20260619 08:23:00.338083 24995 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fb785ee6daa43468ea4795aff8a6773 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb785ee6daa43468ea4795aff8a6773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42613 } }
I20260619 08:23:00.338140 24990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fb785ee6daa43468ea4795aff8a6773 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:23:00.338403 24995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb785ee6daa43468ea4795aff8a6773 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fb785ee6daa43468ea4795aff8a6773. Latest consensus state: current_term: 1 leader_uuid: "6fb785ee6daa43468ea4795aff8a6773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb785ee6daa43468ea4795aff8a6773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42613 } } }
I20260619 08:23:00.338478 24995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb785ee6daa43468ea4795aff8a6773 [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:00.338413 24996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb785ee6daa43468ea4795aff8a6773 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fb785ee6daa43468ea4795aff8a6773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb785ee6daa43468ea4795aff8a6773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42613 } } }
I20260619 08:23:00.339938 24996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb785ee6daa43468ea4795aff8a6773 [sys.catalog]: This master's current role is: LEADER
W20260619 08:23:00.340183 25011 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fb785ee6daa43468ea4795aff8a6773: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:23:00.340245 25011 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:23:00.340296 25000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:23:00.340452 25000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:23:00.341972 25000 catalog_manager.cc:1347] Generated new cluster ID: 7a43e4df44c447bc8325de3658d7e123
I20260619 08:23:00.342044 25000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:23:00 dist-test-slave-d3n8 krb5kdc[24909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 08:23:00.347914 24994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:00.336786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34509 (local address 127.0.0.1:42613)
0619 08:23:00.336985 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:00.336996 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:00.337031 (+ 35us) server_negotiation.cc:407] Connection header received
0619 08:23:00.337143 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:00.337149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:00.337417 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:00.337564 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:00.338471 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:00.340128 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:00.341039 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:00.341627 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:00.341735 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:00.344464 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:00.344468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:00.344469 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:00.344479 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:00.346220 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:00.346675 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:00.346679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:00.346679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:00.346782 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:00.347045 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:00.347046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:00.347047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:00.347314 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:00.347501 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:00.347599 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 08:23:00.347704 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
I20260619 08:23:00.348852 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:23:00.358376 25000 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:23:00.359170 25000 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:23:00.364069 24994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:00.353733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51113 (local address 127.0.0.1:42613)
0619 08:23:00.353773 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:00.353775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:00.357589 (+ 3814us) server_negotiation.cc:407] Connection header received
0619 08:23:00.357663 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:00.357663 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:00.357674 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:00.357750 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:00.358493 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:00.358999 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:00.359794 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:00.360011 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:00.360098 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:00.361346 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:00.361348 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:00.361349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:00.361356 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:00.362857 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:00.363255 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:00.363257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:00.363257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:00.363317 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:00.363553 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:00.363554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:00.363554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:00.363665 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:00.363776 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:00.363893 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 08:23:00.363959 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 08:23:00.369831 25000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fb785ee6daa43468ea4795aff8a6773: Generated new TSK 0
I20260619 08:23:00.370018 25000 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:23:00.375488 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 24922
W20260619 08:23:00.379928 24907 connection.cc:535] client connection to 127.0.0.1:42613 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42613): Connection reset by peer (error 104)
2026-06-19T08:23:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:23:00 dist-test-slave-d3n8 krb5kdc[25021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:23:00 dist-test-slave-d3n8 krb5kdc[25021](info): set up 2 sockets
Jun 19 08:23:00 dist-test-slave-d3n8 krb5kdc[25021](info): commencing operation
krb5kdc: starting...
W20260619 08:23:01.437716 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:23:01 dist-test-slave-d3n8 krb5kdc[25021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:23:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:23:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 08:23:01.569545 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35971
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:23:01.653188 25034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:23:01.653425 25034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:23:01.653450 25034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:23:01.654652 25034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:23:01.654697 25034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:23:01.654711 25034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:23:01.654724 25034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:23:01.654736 25034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:23:01.656253 25034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35971
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:23:01.656522 25034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:23:01.656760 25034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:23:01.661785 25042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:01.661809 25039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:01.661785 25040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:01.662310 25034 server_base.cc:1034] running on GCE node
I20260619 08:23:01.662609 25034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:23:01.662930 25034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:23:01.664127 25034 hybrid_clock.cc:648] HybridClock initialized: now 1781857381664117 us; error 52 us; skew 500 ppm
Jun 19 08:23:01 dist-test-slave-d3n8 krb5kdc[25021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:23:01.666944 25034 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 08:23:01.667497 25034 webserver.cc:460] Webserver started at http://127.0.0.1:45611/ using document root <none> and password file <none>
I20260619 08:23:01.667753 25034 fs_manager.cc:359] Metadata directory not provided
I20260619 08:23:01.667822 25034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:23:01.667989 25034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:23:01.669054 25034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "a6494dee3c274c6097afcf7f8f6b0bdb"
format_stamp: "Formatted at 2026-06-19 08:23:01 on dist-test-slave-d3n8"
server_key: "b370af28ce8c9ef7ec48f57524b8a84a"
server_key_iv: "d4fc24379d3d8f8a341a292b4604cf66"
server_key_version: "encryptionkey@0"
I20260619 08:23:01.669483 25034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "a6494dee3c274c6097afcf7f8f6b0bdb"
format_stamp: "Formatted at 2026-06-19 08:23:01 on dist-test-slave-d3n8"
server_key: "b370af28ce8c9ef7ec48f57524b8a84a"
server_key_iv: "d4fc24379d3d8f8a341a292b4604cf66"
server_key_version: "encryptionkey@0"
I20260619 08:23:01.670828 25034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:23:01.671658 25049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:23:01.671940 25034 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:23:01.672044 25034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "a6494dee3c274c6097afcf7f8f6b0bdb"
format_stamp: "Formatted at 2026-06-19 08:23:01 on dist-test-slave-d3n8"
server_key: "b370af28ce8c9ef7ec48f57524b8a84a"
server_key_iv: "d4fc24379d3d8f8a341a292b4604cf66"
server_key_version: "encryptionkey@0"
I20260619 08:23:01.672120 25034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:23:01.684243 25034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:23:01.686641 25034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:23:01.686823 25034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:23:01.691160 25034 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35971
I20260619 08:23:01.691198 25101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35971 every 8 connection(s)
I20260619 08:23:01.691553 25034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:23:01.692605 25102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:23:01.694792 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 25034
I20260619 08:23:01.694898 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:23:01.695113 25102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6494dee3c274c6097afcf7f8f6b0bdb: Bootstrap starting.
I20260619 08:23:01.695124 4702 external_mini_cluster.cc:1428] Setting key 995a8502e4a6b4ddc662df5f0e928260
I20260619 08:23:01.695744 25102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6494dee3c274c6097afcf7f8f6b0bdb: Neither blocks nor log segments found. Creating new log.
I20260619 08:23:01.696076 25102 log.cc:824] T 00000000000000000000000000000000 P a6494dee3c274c6097afcf7f8f6b0bdb: Log is configured to *not* fsync() on all Append() calls
I20260619 08:23:01.697037 25102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6494dee3c274c6097afcf7f8f6b0bdb: No bootstrap required, opened a new log
I20260619 08:23:01.699398 25102 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6494dee3c274c6097afcf7f8f6b0bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6494dee3c274c6097afcf7f8f6b0bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35971 } }
I20260619 08:23:01.699568 25102 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6494dee3c274c6097afcf7f8f6b0bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:23:01.699604 25102 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6494dee3c274c6097afcf7f8f6b0bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6494dee3c274c6097afcf7f8f6b0bdb, State: Initialized, Role: FOLLOWER
I20260619 08:23:01.699749 25102 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6494dee3c274c6097afcf7f8f6b0bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6494dee3c274c6097afcf7f8f6b0bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35971 } }
I20260619 08:23:01.699841 25102 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6494dee3c274c6097afcf7f8f6b0bdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:23:01.699872 25102 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6494dee3c274c6097afcf7f8f6b0bdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:23:01.699918 25102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6494dee3c274c6097afcf7f8f6b0bdb [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:23:01.700462 25102 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6494dee3c274c6097afcf7f8f6b0bdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6494dee3c274c6097afcf7f8f6b0bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35971 } }
I20260619 08:23:01.700572 25102 leader_election.cc:302] T 00000000000000000000000000000000 P a6494dee3c274c6097afcf7f8f6b0bdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6494dee3c274c6097afcf7f8f6b0bdb; no voters:
I20260619 08:23:01.700779 25102 leader_election.cc:288] T 00000000000000000000000000000000 P a6494dee3c274c6097afcf7f8f6b0bdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:23:01.700802 25107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6494dee3c274c6097afcf7f8f6b0bdb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:23:01.700915 25107 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6494dee3c274c6097afcf7f8f6b0bdb [term 1 LEADER]: Becoming Leader. State: Replica: a6494dee3c274c6097afcf7f8f6b0bdb, State: Running, Role: LEADER
I20260619 08:23:01.700996 25107 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6494dee3c274c6097afcf7f8f6b0bdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6494dee3c274c6097afcf7f8f6b0bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35971 } }
I20260619 08:23:01.701045 25102 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6494dee3c274c6097afcf7f8f6b0bdb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:23:01.701323 25107 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6494dee3c274c6097afcf7f8f6b0bdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6494dee3c274c6097afcf7f8f6b0bdb. Latest consensus state: current_term: 1 leader_uuid: "a6494dee3c274c6097afcf7f8f6b0bdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6494dee3c274c6097afcf7f8f6b0bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35971 } } }
I20260619 08:23:01.701334 25108 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6494dee3c274c6097afcf7f8f6b0bdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6494dee3c274c6097afcf7f8f6b0bdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6494dee3c274c6097afcf7f8f6b0bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35971 } } }
I20260619 08:23:01.701437 25107 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6494dee3c274c6097afcf7f8f6b0bdb [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:01.701483 25108 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6494dee3c274c6097afcf7f8f6b0bdb [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:01.701637 25105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:01.695695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52199 (local address 127.0.0.1:35971)
0619 08:23:01.695962 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:01.695971 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:01.696005 (+ 34us) server_negotiation.cc:407] Connection header received
0619 08:23:01.696520 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:01.696527 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:01.696575 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:01.696738 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:01.696964 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:01.698328 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:01.699093 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:01.699718 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:01.699839 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:01.700674 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:01.700679 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:01.700679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:23:01.700918 (+ 239us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:01.700961 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:23:01.700978 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:01.701213 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:01.701261 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 08:23:01.701289 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":87,"threads_started":1}
I20260619 08:23:01.701735 25113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:23:01.702517 25113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:23:01.702580 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:23:01.704427 25113 catalog_manager.cc:1347] Generated new cluster ID: 59d16402e2924cc69df5d8d3e853ee82
I20260619 08:23:01.704483 25113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:23:01.707118 25105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:01.704059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48629 (local address 127.0.0.1:35971)
0619 08:23:01.704095 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:01.704097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:01.704215 (+ 118us) server_negotiation.cc:407] Connection header received
0619 08:23:01.704261 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:01.704262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:01.704272 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:01.704334 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:01.704541 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:01.705031 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:01.705868 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:01.706093 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:01.706177 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:01.706869 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:01.706870 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:01.706871 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:23:01.706875 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:01.706889 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:23:01.706894 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:01.706948 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:01.707008 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 08:23:01.707018 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 08:23:01.720867 25113 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:23:01.721555 25113 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:23:01.724685 25113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6494dee3c274c6097afcf7f8f6b0bdb: Generated new TSK 0
I20260619 08:23:01.724905 25113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:23:01.727146 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 25034
W20260619 08:23:01.731416 25019 connection.cc:535] client connection to 127.0.0.1:35971 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35971): Connection reset by peer (error 104)
2026-06-19T08:23:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:23:01 dist-test-slave-d3n8 krb5kdc[25133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:23:01 dist-test-slave-d3n8 krb5kdc[25133](info): set up 2 sockets
Jun 19 08:23:01 dist-test-slave-d3n8 krb5kdc[25133](info): commencing operation
krb5kdc: starting...
W20260619 08:23:02.777640 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:23:02 dist-test-slave-d3n8 krb5kdc[25133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:23:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:23:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 08:23:02.905256 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32921
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:23:02.986610 25146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:23:02.986863 25146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:23:02.986901 25146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:23:02.988271 25146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:23:02.988368 25146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:23:02.988395 25146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:23:02.988423 25146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:23:02.988453 25146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:23:02.990147 25146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32921
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:23:02.990442 25146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:23:02.990669 25146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:23:02.995922 25152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:02.995921 25151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:02.995929 25154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:02.996111 25146 server_base.cc:1034] running on GCE node
I20260619 08:23:02.996466 25146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:23:02.996724 25146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:23:02.997946 25146 hybrid_clock.cc:648] HybridClock initialized: now 1781857382997920 us; error 31 us; skew 500 ppm
Jun 19 08:23:02 dist-test-slave-d3n8 krb5kdc[25133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:23:03.000572 25146 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 08:23:03.001046 25146 webserver.cc:460] Webserver started at http://127.0.0.1:45023/ using document root <none> and password file <none>
I20260619 08:23:03.001286 25146 fs_manager.cc:359] Metadata directory not provided
I20260619 08:23:03.001355 25146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:23:03.001509 25146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:23:03.002630 25146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "e523b5ed1fd94af695710b06531cb4a4"
format_stamp: "Formatted at 2026-06-19 08:23:02 on dist-test-slave-d3n8"
server_key: "95a4cbf0090aa2814354373c1de1f386"
server_key_iv: "0e144287018a7301fe1fbdf17d11a97f"
server_key_version: "encryptionkey@0"
I20260619 08:23:03.003012 25146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "e523b5ed1fd94af695710b06531cb4a4"
format_stamp: "Formatted at 2026-06-19 08:23:02 on dist-test-slave-d3n8"
server_key: "95a4cbf0090aa2814354373c1de1f386"
server_key_iv: "0e144287018a7301fe1fbdf17d11a97f"
server_key_version: "encryptionkey@0"
I20260619 08:23:03.004482 25146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:23:03.005363 25161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:23:03.005602 25146 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:23:03.005733 25146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "e523b5ed1fd94af695710b06531cb4a4"
format_stamp: "Formatted at 2026-06-19 08:23:02 on dist-test-slave-d3n8"
server_key: "95a4cbf0090aa2814354373c1de1f386"
server_key_iv: "0e144287018a7301fe1fbdf17d11a97f"
server_key_version: "encryptionkey@0"
I20260619 08:23:03.005815 25146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:23:03.009109 25146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:23:03.011436 25146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:23:03.011601 25146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:23:03.015853 25146 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32921
I20260619 08:23:03.015964 25213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32921 every 8 connection(s)
I20260619 08:23:03.016273 25146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:23:03.017309 25214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:23:03.019778 25214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e523b5ed1fd94af695710b06531cb4a4: Bootstrap starting.
I20260619 08:23:03.020252 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 25146
I20260619 08:23:03.020336 25214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e523b5ed1fd94af695710b06531cb4a4: Neither blocks nor log segments found. Creating new log.
I20260619 08:23:03.020392 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:23:03.020574 4702 external_mini_cluster.cc:1428] Setting key bf8ee1da232088ab697e1d1637cbd9ac
I20260619 08:23:03.020674 25214 log.cc:824] T 00000000000000000000000000000000 P e523b5ed1fd94af695710b06531cb4a4: Log is configured to *not* fsync() on all Append() calls
I20260619 08:23:03.021489 25214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e523b5ed1fd94af695710b06531cb4a4: No bootstrap required, opened a new log
I20260619 08:23:03.023424 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:03.021105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51891 (local address 127.0.0.1:32921)
0619 08:23:03.021313 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:03.021322 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:03.021357 (+ 35us) server_negotiation.cc:407] Connection header received
0619 08:23:03.021920 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:03.021927 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:03.021960 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:03.022118 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:03.022808 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:03.022811 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:03.022811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:23:03.022815 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:03.022871 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:23:03.022882 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:03.022983 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:03.023015 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 08:23:03.023053 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":77,"threads_started":1}
I20260619 08:23:03.023432 25214 raft_consensus.cc:348] T 00000000000000000000000000000000 P e523b5ed1fd94af695710b06531cb4a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e523b5ed1fd94af695710b06531cb4a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32921 } }
I20260619 08:23:03.023648 25214 raft_consensus.cc:374] T 00000000000000000000000000000000 P e523b5ed1fd94af695710b06531cb4a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:23:03.023694 25214 raft_consensus.cc:729] T 00000000000000000000000000000000 P e523b5ed1fd94af695710b06531cb4a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e523b5ed1fd94af695710b06531cb4a4, State: Initialized, Role: FOLLOWER
I20260619 08:23:03.023830 25214 consensus_queue.cc:260] T 00000000000000000000000000000000 P e523b5ed1fd94af695710b06531cb4a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e523b5ed1fd94af695710b06531cb4a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32921 } }
I20260619 08:23:03.023933 25214 raft_consensus.cc:388] T 00000000000000000000000000000000 P e523b5ed1fd94af695710b06531cb4a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:23:03.023980 25214 raft_consensus.cc:482] T 00000000000000000000000000000000 P e523b5ed1fd94af695710b06531cb4a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:23:03.024029 25214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e523b5ed1fd94af695710b06531cb4a4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:23:03.024411 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:23:03.024716 25214 raft_consensus.cc:504] T 00000000000000000000000000000000 P e523b5ed1fd94af695710b06531cb4a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e523b5ed1fd94af695710b06531cb4a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32921 } }
I20260619 08:23:03.024814 25214 leader_election.cc:302] T 00000000000000000000000000000000 P e523b5ed1fd94af695710b06531cb4a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e523b5ed1fd94af695710b06531cb4a4; no voters:
I20260619 08:23:03.024994 25214 leader_election.cc:288] T 00000000000000000000000000000000 P e523b5ed1fd94af695710b06531cb4a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:23:03.025020 25219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e523b5ed1fd94af695710b06531cb4a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:23:03.025138 25219 raft_consensus.cc:686] T 00000000000000000000000000000000 P e523b5ed1fd94af695710b06531cb4a4 [term 1 LEADER]: Becoming Leader. State: Replica: e523b5ed1fd94af695710b06531cb4a4, State: Running, Role: LEADER
I20260619 08:23:03.025241 25214 sys_catalog.cc:564] T 00000000000000000000000000000000 P e523b5ed1fd94af695710b06531cb4a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:23:03.025287 25219 consensus_queue.cc:237] T 00000000000000000000000000000000 P e523b5ed1fd94af695710b06531cb4a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e523b5ed1fd94af695710b06531cb4a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32921 } }
I20260619 08:23:03.025763 25221 sys_catalog.cc:455] T 00000000000000000000000000000000 P e523b5ed1fd94af695710b06531cb4a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e523b5ed1fd94af695710b06531cb4a4. Latest consensus state: current_term: 1 leader_uuid: "e523b5ed1fd94af695710b06531cb4a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e523b5ed1fd94af695710b06531cb4a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32921 } } }
I20260619 08:23:03.025868 25221 sys_catalog.cc:458] T 00000000000000000000000000000000 P e523b5ed1fd94af695710b06531cb4a4 [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:03.026054 25220 sys_catalog.cc:455] T 00000000000000000000000000000000 P e523b5ed1fd94af695710b06531cb4a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e523b5ed1fd94af695710b06531cb4a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e523b5ed1fd94af695710b06531cb4a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32921 } } }
I20260619 08:23:03.026110 25220 sys_catalog.cc:458] T 00000000000000000000000000000000 P e523b5ed1fd94af695710b06531cb4a4 [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:03.026273 25233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:23:03.026798 25233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:23:03.028287 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:03.026109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52025 (local address 127.0.0.1:32921)
0619 08:23:03.026143 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:03.026145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:03.026153 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:23:03.026172 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:03.026173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:03.026178 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:03.026238 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:03.028032 (+ 1794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:03.028034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:03.028034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:23:03.028038 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:03.028053 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:23:03.028057 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:03.028103 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:03.028207 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 08:23:03.028221 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 08:23:03.028601 25233 catalog_manager.cc:1347] Generated new cluster ID: b504ed2b546c48909a9b54cdc87bef10
I20260619 08:23:03.028687 25233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:23:03.035472 25233 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:23:03.035630 25233 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:23:03.038580 25233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e523b5ed1fd94af695710b06531cb4a4: Generated new TSK 0
I20260619 08:23:03.038765 25233 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:23:03.043779 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 25146
2026-06-19T08:23:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:23:03 dist-test-slave-d3n8 krb5kdc[25245](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:23:03 dist-test-slave-d3n8 krb5kdc[25245](info): set up 2 sockets
Jun 19 08:23:03 dist-test-slave-d3n8 krb5kdc[25245](info): commencing operation
krb5kdc: starting...
W20260619 08:23:04.109750 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:23:04 dist-test-slave-d3n8 krb5kdc[25245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:23:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:23:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 08:23:04.237512 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36203
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:23:04.318020 25258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:23:04.318274 25258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:23:04.318313 25258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:23:04.319458 25258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:23:04.319540 25258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:23:04.319568 25258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:23:04.319598 25258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:23:04.319620 25258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:23:04.321071 25258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:23:04.321345 25258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:23:04.321591 25258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:23:04.326479 25266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:04.326480 25264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:04.326498 25263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:04.326862 25258 server_base.cc:1034] running on GCE node
I20260619 08:23:04.327116 25258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:23:04.327419 25258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:23:04.328590 25258 hybrid_clock.cc:648] HybridClock initialized: now 1781857384328576 us; error 28 us; skew 500 ppm
Jun 19 08:23:04 dist-test-slave-d3n8 krb5kdc[25245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:23:04.331233 25258 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 08:23:04.331760 25258 webserver.cc:460] Webserver started at http://127.0.0.1:35455/ using document root <none> and password file <none>
I20260619 08:23:04.332036 25258 fs_manager.cc:359] Metadata directory not provided
I20260619 08:23:04.332109 25258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:23:04.332268 25258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:23:04.333737 25258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "be08d3e5af48450db15e39efa0935343"
format_stamp: "Formatted at 2026-06-19 08:23:04 on dist-test-slave-d3n8"
server_key: "3a75327c8cef211580891a35f767200f"
server_key_iv: "8d6665dfeaeffac39cf733e8230d2b95"
server_key_version: "encryptionkey@0"
I20260619 08:23:04.334129 25258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "be08d3e5af48450db15e39efa0935343"
format_stamp: "Formatted at 2026-06-19 08:23:04 on dist-test-slave-d3n8"
server_key: "3a75327c8cef211580891a35f767200f"
server_key_iv: "8d6665dfeaeffac39cf733e8230d2b95"
server_key_version: "encryptionkey@0"
I20260619 08:23:04.335537 25258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:23:04.336352 25273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:23:04.336534 25258 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:23:04.336645 25258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "be08d3e5af48450db15e39efa0935343"
format_stamp: "Formatted at 2026-06-19 08:23:04 on dist-test-slave-d3n8"
server_key: "3a75327c8cef211580891a35f767200f"
server_key_iv: "8d6665dfeaeffac39cf733e8230d2b95"
server_key_version: "encryptionkey@0"
I20260619 08:23:04.336711 25258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:23:04.354339 25258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:23:04.356818 25258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:23:04.356995 25258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:23:04.361322 25258 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36203
I20260619 08:23:04.361358 25325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36203 every 8 connection(s)
I20260619 08:23:04.361773 25258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:23:04.362484 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 25258
I20260619 08:23:04.362607 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:23:04.362816 4702 external_mini_cluster.cc:1428] Setting key 105f1856a6c50b3faaa3301fdd4d0a25
I20260619 08:23:04.362939 25326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:23:04.366379 25326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be08d3e5af48450db15e39efa0935343: Bootstrap starting.
I20260619 08:23:04.367062 25326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be08d3e5af48450db15e39efa0935343: Neither blocks nor log segments found. Creating new log.
I20260619 08:23:04.367363 25326 log.cc:824] T 00000000000000000000000000000000 P be08d3e5af48450db15e39efa0935343: Log is configured to *not* fsync() on all Append() calls
I20260619 08:23:04.368238 25326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be08d3e5af48450db15e39efa0935343: No bootstrap required, opened a new log
Jun 19 08:23:04 dist-test-slave-d3n8 krb5kdc[25245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 08:23:04.371024 25326 raft_consensus.cc:348] T 00000000000000000000000000000000 P be08d3e5af48450db15e39efa0935343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be08d3e5af48450db15e39efa0935343" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36203 } }
I20260619 08:23:04.371238 25326 raft_consensus.cc:374] T 00000000000000000000000000000000 P be08d3e5af48450db15e39efa0935343 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:23:04.371306 25326 raft_consensus.cc:729] T 00000000000000000000000000000000 P be08d3e5af48450db15e39efa0935343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be08d3e5af48450db15e39efa0935343, State: Initialized, Role: FOLLOWER
I20260619 08:23:04.371454 25326 consensus_queue.cc:260] T 00000000000000000000000000000000 P be08d3e5af48450db15e39efa0935343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be08d3e5af48450db15e39efa0935343" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36203 } }
I20260619 08:23:04.371542 25326 raft_consensus.cc:388] T 00000000000000000000000000000000 P be08d3e5af48450db15e39efa0935343 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:23:04.371595 25326 raft_consensus.cc:482] T 00000000000000000000000000000000 P be08d3e5af48450db15e39efa0935343 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:23:04.371758 25326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be08d3e5af48450db15e39efa0935343 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:23:04.372517 25326 raft_consensus.cc:504] T 00000000000000000000000000000000 P be08d3e5af48450db15e39efa0935343 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be08d3e5af48450db15e39efa0935343" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36203 } }
I20260619 08:23:04.372653 25326 leader_election.cc:302] T 00000000000000000000000000000000 P be08d3e5af48450db15e39efa0935343 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be08d3e5af48450db15e39efa0935343; no voters:
I20260619 08:23:04.372838 25326 leader_election.cc:288] T 00000000000000000000000000000000 P be08d3e5af48450db15e39efa0935343 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:23:04.372910 25331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be08d3e5af48450db15e39efa0935343 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:23:04.373068 25331 raft_consensus.cc:686] T 00000000000000000000000000000000 P be08d3e5af48450db15e39efa0935343 [term 1 LEADER]: Becoming Leader. State: Replica: be08d3e5af48450db15e39efa0935343, State: Running, Role: LEADER
I20260619 08:23:04.373172 25326 sys_catalog.cc:564] T 00000000000000000000000000000000 P be08d3e5af48450db15e39efa0935343 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:23:04.373268 25331 consensus_queue.cc:237] T 00000000000000000000000000000000 P be08d3e5af48450db15e39efa0935343 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be08d3e5af48450db15e39efa0935343" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36203 } }
I20260619 08:23:04.373664 25331 sys_catalog.cc:455] T 00000000000000000000000000000000 P be08d3e5af48450db15e39efa0935343 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be08d3e5af48450db15e39efa0935343. Latest consensus state: current_term: 1 leader_uuid: "be08d3e5af48450db15e39efa0935343" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be08d3e5af48450db15e39efa0935343" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36203 } } }
I20260619 08:23:04.373673 25333 sys_catalog.cc:455] T 00000000000000000000000000000000 P be08d3e5af48450db15e39efa0935343 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be08d3e5af48450db15e39efa0935343" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be08d3e5af48450db15e39efa0935343" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36203 } } }
I20260619 08:23:04.373760 25331 sys_catalog.cc:458] T 00000000000000000000000000000000 P be08d3e5af48450db15e39efa0935343 [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:04.373795 25333 sys_catalog.cc:458] T 00000000000000000000000000000000 P be08d3e5af48450db15e39efa0935343 [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:04.374890 25329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:04.363472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34447 (local address 127.0.0.1:36203)
0619 08:23:04.363643 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:04.363650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:04.363677 (+ 27us) server_negotiation.cc:407] Connection header received
0619 08:23:04.364097 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:04.364103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:04.364148 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:04.364258 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:04.365087 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:04.366116 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:04.367179 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:04.367677 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:04.367819 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:04.370584 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:04.370588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:04.370589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:04.370602 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:04.372444 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:04.372928 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:04.372932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:04.372932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:04.373064 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:04.373320 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:04.373321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:04.373321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:04.373720 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:04.374325 (+ 605us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:04.374418 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 08:23:04.374576 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
W20260619 08:23:04.375177 25347 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be08d3e5af48450db15e39efa0935343: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:23:04.375237 25347 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:23:04.375288 25338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:23:04.375458 25338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:23:04.375897 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:23:04.377192 25338 catalog_manager.cc:1347] Generated new cluster ID: f8cefa8f98754ac9b51e31b362de3a69
I20260619 08:23:04.377250 25338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:23:04.384303 25329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:04.377479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53367 (local address 127.0.0.1:36203)
0619 08:23:04.377529 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:04.377531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:04.377654 (+ 123us) server_negotiation.cc:407] Connection header received
0619 08:23:04.377716 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:04.377717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:04.377725 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:04.377770 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:04.378376 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:04.378878 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:04.379846 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:04.380054 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:04.380116 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:04.381441 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:04.381442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:04.381442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:04.381449 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:04.382964 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:04.383357 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:04.383358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:04.383359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:04.383439 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:04.383648 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:04.383649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:04.383649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:04.383770 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:04.383963 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:04.384087 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 08:23:04.384190 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 08:23:04.394969 25338 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:23:04.395615 25338 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:23:04.407166 25338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be08d3e5af48450db15e39efa0935343: Generated new TSK 0
I20260619 08:23:04.407410 25338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:23:04.414894 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 25258
2026-06-19T08:23:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:23:04 dist-test-slave-d3n8 krb5kdc[25357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:23:04 dist-test-slave-d3n8 krb5kdc[25357](info): set up 2 sockets
Jun 19 08:23:04 dist-test-slave-d3n8 krb5kdc[25357](info): commencing operation
krb5kdc: starting...
W20260619 08:23:05.465507 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:23:05 dist-test-slave-d3n8 krb5kdc[25357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:23:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:23:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 08:23:05.598192 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40787
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:23:05.678857 25370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:23:05.679060 25370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:23:05.679085 25370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:23:05.680310 25370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:23:05.680352 25370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:23:05.680367 25370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:23:05.680379 25370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:23:05.680392 25370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:23:05.681887 25370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:23:05.682139 25370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:23:05.682369 25370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:23:05.687763 25375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:05.687722 25378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:05.687722 25376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:05.687889 25370 server_base.cc:1034] running on GCE node
I20260619 08:23:05.688207 25370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:23:05.688503 25370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:23:05.689677 25370 hybrid_clock.cc:648] HybridClock initialized: now 1781857385689663 us; error 66 us; skew 500 ppm
Jun 19 08:23:05 dist-test-slave-d3n8 krb5kdc[25357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:23:05.692312 25370 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 08:23:05.692819 25370 webserver.cc:460] Webserver started at http://127.0.0.1:35001/ using document root <none> and password file <none>
I20260619 08:23:05.693066 25370 fs_manager.cc:359] Metadata directory not provided
I20260619 08:23:05.693136 25370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:23:05.693280 25370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:23:05.694445 25370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "48bbca974eb344e990f444f60a1fe04a"
format_stamp: "Formatted at 2026-06-19 08:23:05 on dist-test-slave-d3n8"
server_key: "683f44c857dc2a88dcf43714aa2f96d7"
server_key_iv: "2aea4b4cfd47d2c55976238ee6a05576"
server_key_version: "encryptionkey@0"
I20260619 08:23:05.694823 25370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "48bbca974eb344e990f444f60a1fe04a"
format_stamp: "Formatted at 2026-06-19 08:23:05 on dist-test-slave-d3n8"
server_key: "683f44c857dc2a88dcf43714aa2f96d7"
server_key_iv: "2aea4b4cfd47d2c55976238ee6a05576"
server_key_version: "encryptionkey@0"
I20260619 08:23:05.696177 25370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:23:05.696949 25385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:23:05.697147 25370 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:23:05.697248 25370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "48bbca974eb344e990f444f60a1fe04a"
format_stamp: "Formatted at 2026-06-19 08:23:05 on dist-test-slave-d3n8"
server_key: "683f44c857dc2a88dcf43714aa2f96d7"
server_key_iv: "2aea4b4cfd47d2c55976238ee6a05576"
server_key_version: "encryptionkey@0"
I20260619 08:23:05.697336 25370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:23:05.720396 25370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:23:05.722900 25370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:23:05.723083 25370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:23:05.727033 25370 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40787
I20260619 08:23:05.727093 25437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40787 every 8 connection(s)
I20260619 08:23:05.727452 25370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:23:05.728634 25438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:23:05.731073 25438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48bbca974eb344e990f444f60a1fe04a: Bootstrap starting.
I20260619 08:23:05.731653 25438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48bbca974eb344e990f444f60a1fe04a: Neither blocks nor log segments found. Creating new log.
I20260619 08:23:05.731953 25438 log.cc:824] T 00000000000000000000000000000000 P 48bbca974eb344e990f444f60a1fe04a: Log is configured to *not* fsync() on all Append() calls
I20260619 08:23:05.732703 25438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48bbca974eb344e990f444f60a1fe04a: No bootstrap required, opened a new log
I20260619 08:23:05.733348 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 25370
I20260619 08:23:05.733496 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:23:05.733705 4702 external_mini_cluster.cc:1428] Setting key 42156ee27df600a2f6de1d3e8005bcfd
I20260619 08:23:05.735035 25438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48bbca974eb344e990f444f60a1fe04a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48bbca974eb344e990f444f60a1fe04a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40787 } }
I20260619 08:23:05.735235 25438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48bbca974eb344e990f444f60a1fe04a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:23:05.735283 25438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48bbca974eb344e990f444f60a1fe04a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48bbca974eb344e990f444f60a1fe04a, State: Initialized, Role: FOLLOWER
I20260619 08:23:05.735419 25438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48bbca974eb344e990f444f60a1fe04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48bbca974eb344e990f444f60a1fe04a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40787 } }
I20260619 08:23:05.735512 25438 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48bbca974eb344e990f444f60a1fe04a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:23:05.735571 25438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48bbca974eb344e990f444f60a1fe04a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:23:05.735623 25438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48bbca974eb344e990f444f60a1fe04a [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:23:05.736230 25438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48bbca974eb344e990f444f60a1fe04a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48bbca974eb344e990f444f60a1fe04a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40787 } }
I20260619 08:23:05.736320 25438 leader_election.cc:302] T 00000000000000000000000000000000 P 48bbca974eb344e990f444f60a1fe04a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48bbca974eb344e990f444f60a1fe04a; no voters:
I20260619 08:23:05.736531 25438 leader_election.cc:288] T 00000000000000000000000000000000 P 48bbca974eb344e990f444f60a1fe04a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:23:05.736555 25443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48bbca974eb344e990f444f60a1fe04a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:23:05.736666 25443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48bbca974eb344e990f444f60a1fe04a [term 1 LEADER]: Becoming Leader. State: Replica: 48bbca974eb344e990f444f60a1fe04a, State: Running, Role: LEADER
I20260619 08:23:05.736752 25443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48bbca974eb344e990f444f60a1fe04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48bbca974eb344e990f444f60a1fe04a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40787 } }
I20260619 08:23:05.736812 25438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48bbca974eb344e990f444f60a1fe04a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:23:05.737097 25443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48bbca974eb344e990f444f60a1fe04a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48bbca974eb344e990f444f60a1fe04a. Latest consensus state: current_term: 1 leader_uuid: "48bbca974eb344e990f444f60a1fe04a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48bbca974eb344e990f444f60a1fe04a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40787 } } }
I20260619 08:23:05.737177 25443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48bbca974eb344e990f444f60a1fe04a [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:05.737111 25444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48bbca974eb344e990f444f60a1fe04a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48bbca974eb344e990f444f60a1fe04a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48bbca974eb344e990f444f60a1fe04a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40787 } } }
I20260619 08:23:05.737353 25444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48bbca974eb344e990f444f60a1fe04a [sys.catalog]: This master's current role is: LEADER
W20260619 08:23:05.738703 25459 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 48bbca974eb344e990f444f60a1fe04a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:23:05.738759 25459 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:23:05.738802 25448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:23:05.738942 25448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:23:05.740392 25442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:05.734270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38235 (local address 127.0.0.1:40787)
0619 08:23:05.734469 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:05.734482 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:05.734508 (+ 26us) server_negotiation.cc:407] Connection header received
0619 08:23:05.735001 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:05.735009 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:05.735063 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:05.735179 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:05.735461 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:05.736987 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:05.738145 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:05.738695 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:05.738820 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:05.739581 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:05.739585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:05.739586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:23:05.739785 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:05.739816 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:23:05.739828 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:05.739918 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:05.740152 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 08:23:05.740224 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":95,"threads_started":1}
I20260619 08:23:05.740867 25448 catalog_manager.cc:1347] Generated new cluster ID: 0414ee8c9e8c428fa598f683c4101cf8
I20260619 08:23:05.740926 25448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:23:05.741406 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:23:05.745636 25442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:05.742741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51471 (local address 127.0.0.1:40787)
0619 08:23:05.742779 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:05.742781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:05.742904 (+ 123us) server_negotiation.cc:407] Connection header received
0619 08:23:05.742977 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:05.742978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:05.742989 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:05.743034 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:05.743177 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:05.743629 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:05.744335 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:05.744551 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:05.744624 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:05.745325 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:05.745326 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:05.745327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:23:05.745332 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:05.745346 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:23:05.745351 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:05.745427 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:05.745503 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 08:23:05.745526 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:23:05.746434 25448 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:23:05.746954 25448 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:23:05.760550 25448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48bbca974eb344e990f444f60a1fe04a: Generated new TSK 0
I20260619 08:23:05.760775 25448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:23:05.764508 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 25370
2026-06-19T08:23:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1349 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:23:05 dist-test-slave-d3n8 krb5kdc[25469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:23:05 dist-test-slave-d3n8 krb5kdc[25469](info): set up 2 sockets
Jun 19 08:23:05 dist-test-slave-d3n8 krb5kdc[25469](info): commencing operation
krb5kdc: starting...
W20260619 08:23:06.809542 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:23:06 dist-test-slave-d3n8 krb5kdc[25469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:23:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:23:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 08:23:06.937203 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39511
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:23:07.026526 25482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:23:07.026763 25482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:23:07.026788 25482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:23:07.028016 25482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:23:07.028067 25482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:23:07.028090 25482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:23:07.028102 25482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:23:07.028115 25482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:23:07.029685 25482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:23:07.029949 25482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:23:07.030189 25482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:23:07.035125 25490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:07.035168 25487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:07.035168 25488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:07.035475 25482 server_base.cc:1034] running on GCE node
I20260619 08:23:07.035763 25482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:23:07.036069 25482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:23:07.037266 25482 hybrid_clock.cc:648] HybridClock initialized: now 1781857387037255 us; error 42 us; skew 500 ppm
Jun 19 08:23:07 dist-test-slave-d3n8 krb5kdc[25469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:23:07.046198 25482 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 08:23:07.046746 25482 webserver.cc:460] Webserver started at http://127.0.0.1:41491/ using document root <none> and password file <none>
I20260619 08:23:07.046991 25482 fs_manager.cc:359] Metadata directory not provided
I20260619 08:23:07.047060 25482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:23:07.047230 25482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:23:07.048328 25482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "daaaca927b2c41608d6b3a1094a1ff69"
format_stamp: "Formatted at 2026-06-19 08:23:07 on dist-test-slave-d3n8"
server_key: "361d0556dd63f8698eb1a4236bfa3049"
server_key_iv: "0f9b4725239161a1c159040fb6a0b6c5"
server_key_version: "encryptionkey@0"
I20260619 08:23:07.048709 25482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "daaaca927b2c41608d6b3a1094a1ff69"
format_stamp: "Formatted at 2026-06-19 08:23:07 on dist-test-slave-d3n8"
server_key: "361d0556dd63f8698eb1a4236bfa3049"
server_key_iv: "0f9b4725239161a1c159040fb6a0b6c5"
server_key_version: "encryptionkey@0"
I20260619 08:23:07.050319 25482 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:23:07.051187 25497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:23:07.051417 25482 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:23:07.051527 25482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "daaaca927b2c41608d6b3a1094a1ff69"
format_stamp: "Formatted at 2026-06-19 08:23:07 on dist-test-slave-d3n8"
server_key: "361d0556dd63f8698eb1a4236bfa3049"
server_key_iv: "0f9b4725239161a1c159040fb6a0b6c5"
server_key_version: "encryptionkey@0"
I20260619 08:23:07.051608 25482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:23:07.055188 25482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:23:07.057673 25482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:23:07.057848 25482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:23:07.062285 25482 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39511
I20260619 08:23:07.062335 25549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39511 every 8 connection(s)
I20260619 08:23:07.062677 25482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:23:07.063717 25550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:23:07.066195 25550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daaaca927b2c41608d6b3a1094a1ff69: Bootstrap starting.
I20260619 08:23:07.066818 25550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daaaca927b2c41608d6b3a1094a1ff69: Neither blocks nor log segments found. Creating new log.
I20260619 08:23:07.067113 25550 log.cc:824] T 00000000000000000000000000000000 P daaaca927b2c41608d6b3a1094a1ff69: Log is configured to *not* fsync() on all Append() calls
I20260619 08:23:07.067879 25550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daaaca927b2c41608d6b3a1094a1ff69: No bootstrap required, opened a new log
I20260619 08:23:07.069581 25550 raft_consensus.cc:348] T 00000000000000000000000000000000 P daaaca927b2c41608d6b3a1094a1ff69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daaaca927b2c41608d6b3a1094a1ff69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39511 } }
I20260619 08:23:07.069729 25550 raft_consensus.cc:374] T 00000000000000000000000000000000 P daaaca927b2c41608d6b3a1094a1ff69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:23:07.069753 25550 raft_consensus.cc:729] T 00000000000000000000000000000000 P daaaca927b2c41608d6b3a1094a1ff69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daaaca927b2c41608d6b3a1094a1ff69, State: Initialized, Role: FOLLOWER
I20260619 08:23:07.069911 25550 consensus_queue.cc:260] T 00000000000000000000000000000000 P daaaca927b2c41608d6b3a1094a1ff69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daaaca927b2c41608d6b3a1094a1ff69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39511 } }
I20260619 08:23:07.069993 25550 raft_consensus.cc:388] T 00000000000000000000000000000000 P daaaca927b2c41608d6b3a1094a1ff69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:23:07.070019 25550 raft_consensus.cc:482] T 00000000000000000000000000000000 P daaaca927b2c41608d6b3a1094a1ff69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:23:07.070072 25550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daaaca927b2c41608d6b3a1094a1ff69 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:23:07.070653 25550 raft_consensus.cc:504] T 00000000000000000000000000000000 P daaaca927b2c41608d6b3a1094a1ff69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daaaca927b2c41608d6b3a1094a1ff69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39511 } }
I20260619 08:23:07.070747 25550 leader_election.cc:302] T 00000000000000000000000000000000 P daaaca927b2c41608d6b3a1094a1ff69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daaaca927b2c41608d6b3a1094a1ff69; no voters:
I20260619 08:23:07.070919 25550 leader_election.cc:288] T 00000000000000000000000000000000 P daaaca927b2c41608d6b3a1094a1ff69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:23:07.071094 25554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daaaca927b2c41608d6b3a1094a1ff69 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:23:07.071187 25550 sys_catalog.cc:564] T 00000000000000000000000000000000 P daaaca927b2c41608d6b3a1094a1ff69 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:23:07.071264 25554 raft_consensus.cc:686] T 00000000000000000000000000000000 P daaaca927b2c41608d6b3a1094a1ff69 [term 1 LEADER]: Becoming Leader. State: Replica: daaaca927b2c41608d6b3a1094a1ff69, State: Running, Role: LEADER
I20260619 08:23:07.071408 25554 consensus_queue.cc:237] T 00000000000000000000000000000000 P daaaca927b2c41608d6b3a1094a1ff69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daaaca927b2c41608d6b3a1094a1ff69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39511 } }
I20260619 08:23:07.072108 25555 sys_catalog.cc:455] T 00000000000000000000000000000000 P daaaca927b2c41608d6b3a1094a1ff69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader daaaca927b2c41608d6b3a1094a1ff69. Latest consensus state: current_term: 1 leader_uuid: "daaaca927b2c41608d6b3a1094a1ff69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daaaca927b2c41608d6b3a1094a1ff69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39511 } } }
I20260619 08:23:07.072266 25555 sys_catalog.cc:458] T 00000000000000000000000000000000 P daaaca927b2c41608d6b3a1094a1ff69 [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:07.072291 25553 sys_catalog.cc:455] T 00000000000000000000000000000000 P daaaca927b2c41608d6b3a1094a1ff69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "daaaca927b2c41608d6b3a1094a1ff69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daaaca927b2c41608d6b3a1094a1ff69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39511 } } }
I20260619 08:23:07.072384 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 25482
I20260619 08:23:07.072440 25553 sys_catalog.cc:458] T 00000000000000000000000000000000 P daaaca927b2c41608d6b3a1094a1ff69 [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:07.072506 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:23:07.072662 4702 external_mini_cluster.cc:1428] Setting key 1c372f7cf749d243a49b8e0941d01a63
I20260619 08:23:07.073179 25564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:23:07.073819 25564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:23:07.075475 25564 catalog_manager.cc:1347] Generated new cluster ID: ca9bfbfc1dbe43f0801a8814c84097e1
I20260619 08:23:07.075532 25564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:23:07.076402 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:07.073531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59805 (local address 127.0.0.1:39511)
0619 08:23:07.074038 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:07.074046 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:07.074091 (+ 45us) server_negotiation.cc:407] Connection header received
0619 08:23:07.074483 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:07.074493 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:07.074528 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:07.074682 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:07.075984 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:07.075987 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:07.075987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:23:07.075992 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:07.076043 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:23:07.076054 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:07.076124 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:07.076207 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 08:23:07.076249 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":127,"threads_started":1}
I20260619 08:23:07.077466 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:23:07.080039 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:07.078911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40101 (local address 127.0.0.1:39511)
0619 08:23:07.078949 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:07.078950 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:07.079114 (+ 164us) server_negotiation.cc:407] Connection header received
0619 08:23:07.079140 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:07.079141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:07.079147 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:07.079197 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:07.079815 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:07.079816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:07.079816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:23:07.079821 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:07.079836 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:23:07.079841 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:07.079886 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:07.079939 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 08:23:07.079955 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 08:23:07.084918 25564 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:23:07.085141 25564 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:23:07.092442 25564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P daaaca927b2c41608d6b3a1094a1ff69: Generated new TSK 0
I20260619 08:23:07.092669 25564 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:23:07.099911 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 25482
2026-06-19T08:23:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:23:07 dist-test-slave-d3n8 krb5kdc[25581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:23:07 dist-test-slave-d3n8 krb5kdc[25581](info): set up 2 sockets
Jun 19 08:23:07 dist-test-slave-d3n8 krb5kdc[25581](info): commencing operation
krb5kdc: starting...
W20260619 08:23:08.182096 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:23:08 dist-test-slave-d3n8 krb5kdc[25581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:23:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:23:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 08:23:08.308323 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36879
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:23:08.399390 25594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:23:08.399633 25594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:23:08.399658 25594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:23:08.400928 25594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:23:08.400979 25594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:23:08.400997 25594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:23:08.401010 25594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:23:08.401022 25594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:23:08.402563 25594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:23:08.402832 25594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:23:08.403074 25594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:23:08.408319 25599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:08.408459 25602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:08.408504 25594 server_base.cc:1034] running on GCE node
W20260619 08:23:08.408425 25600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:08.408911 25594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:23:08.409250 25594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:23:08.410506 25594 hybrid_clock.cc:648] HybridClock initialized: now 1781857388410463 us; error 59 us; skew 500 ppm
Jun 19 08:23:08 dist-test-slave-d3n8 krb5kdc[25581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:23:08.413118 25594 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 08:23:08.413673 25594 webserver.cc:460] Webserver started at http://127.0.0.1:43171/ using document root <none> and password file <none>
I20260619 08:23:08.413904 25594 fs_manager.cc:359] Metadata directory not provided
I20260619 08:23:08.413971 25594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:23:08.414137 25594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:23:08.415223 25594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "873b4fae16c14b19824e2c420e4b4a3d"
format_stamp: "Formatted at 2026-06-19 08:23:08 on dist-test-slave-d3n8"
server_key: "4823bb40a18e3eb7d833edabb4f3a08f"
server_key_iv: "743928995aef2e5ff6930ccb4af8fb3e"
server_key_version: "encryptionkey@0"
I20260619 08:23:08.415597 25594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "873b4fae16c14b19824e2c420e4b4a3d"
format_stamp: "Formatted at 2026-06-19 08:23:08 on dist-test-slave-d3n8"
server_key: "4823bb40a18e3eb7d833edabb4f3a08f"
server_key_iv: "743928995aef2e5ff6930ccb4af8fb3e"
server_key_version: "encryptionkey@0"
I20260619 08:23:08.417044 25594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:23:08.417936 25609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:23:08.418124 25594 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 08:23:08.418231 25594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "873b4fae16c14b19824e2c420e4b4a3d"
format_stamp: "Formatted at 2026-06-19 08:23:08 on dist-test-slave-d3n8"
server_key: "4823bb40a18e3eb7d833edabb4f3a08f"
server_key_iv: "743928995aef2e5ff6930ccb4af8fb3e"
server_key_version: "encryptionkey@0"
I20260619 08:23:08.418318 25594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:23:08.431563 25594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:23:08.434005 25594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:23:08.434149 25594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:23:08.438197 25594 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36879
I20260619 08:23:08.438248 25661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36879 every 8 connection(s)
I20260619 08:23:08.438577 25594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:23:08.439663 25662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:23:08.441951 25662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 873b4fae16c14b19824e2c420e4b4a3d: Bootstrap starting.
I20260619 08:23:08.442497 25662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 873b4fae16c14b19824e2c420e4b4a3d: Neither blocks nor log segments found. Creating new log.
I20260619 08:23:08.442731 25662 log.cc:824] T 00000000000000000000000000000000 P 873b4fae16c14b19824e2c420e4b4a3d: Log is configured to *not* fsync() on all Append() calls
I20260619 08:23:08.443418 25662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 873b4fae16c14b19824e2c420e4b4a3d: No bootstrap required, opened a new log
I20260619 08:23:08.444931 25662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 873b4fae16c14b19824e2c420e4b4a3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "873b4fae16c14b19824e2c420e4b4a3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36879 } }
I20260619 08:23:08.445065 25662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 873b4fae16c14b19824e2c420e4b4a3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:23:08.445101 25662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 873b4fae16c14b19824e2c420e4b4a3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873b4fae16c14b19824e2c420e4b4a3d, State: Initialized, Role: FOLLOWER
I20260619 08:23:08.445236 25662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 873b4fae16c14b19824e2c420e4b4a3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "873b4fae16c14b19824e2c420e4b4a3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36879 } }
I20260619 08:23:08.445326 25662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 873b4fae16c14b19824e2c420e4b4a3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:23:08.445349 25662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 873b4fae16c14b19824e2c420e4b4a3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:23:08.445427 25662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 873b4fae16c14b19824e2c420e4b4a3d [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:23:08.445911 25662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 873b4fae16c14b19824e2c420e4b4a3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "873b4fae16c14b19824e2c420e4b4a3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36879 } }
I20260619 08:23:08.445992 25662 leader_election.cc:302] T 00000000000000000000000000000000 P 873b4fae16c14b19824e2c420e4b4a3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 873b4fae16c14b19824e2c420e4b4a3d; no voters:
I20260619 08:23:08.446125 25662 leader_election.cc:288] T 00000000000000000000000000000000 P 873b4fae16c14b19824e2c420e4b4a3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:23:08.446187 25665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 873b4fae16c14b19824e2c420e4b4a3d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:23:08.446410 25662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 873b4fae16c14b19824e2c420e4b4a3d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:23:08.446424 25665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 873b4fae16c14b19824e2c420e4b4a3d [term 1 LEADER]: Becoming Leader. State: Replica: 873b4fae16c14b19824e2c420e4b4a3d, State: Running, Role: LEADER
I20260619 08:23:08.446568 25665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 873b4fae16c14b19824e2c420e4b4a3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "873b4fae16c14b19824e2c420e4b4a3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36879 } }
I20260619 08:23:08.447167 25666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 873b4fae16c14b19824e2c420e4b4a3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "873b4fae16c14b19824e2c420e4b4a3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "873b4fae16c14b19824e2c420e4b4a3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36879 } } }
I20260619 08:23:08.447268 25666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 873b4fae16c14b19824e2c420e4b4a3d [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:08.447283 25667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 873b4fae16c14b19824e2c420e4b4a3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 873b4fae16c14b19824e2c420e4b4a3d. Latest consensus state: current_term: 1 leader_uuid: "873b4fae16c14b19824e2c420e4b4a3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "873b4fae16c14b19824e2c420e4b4a3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36879 } } }
I20260619 08:23:08.447365 25667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 873b4fae16c14b19824e2c420e4b4a3d [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:08.447584 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 25594
I20260619 08:23:08.447726 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:23:08.447926 4702 external_mini_cluster.cc:1428] Setting key 6209916a8ba4149df219c7819ed98aa5
I20260619 08:23:08.448096 25674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:23:08.448604 25674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:23:08.450189 25674 catalog_manager.cc:1347] Generated new cluster ID: 399d04ffc3a74d6686e31a6b04e2c55f
I20260619 08:23:08.450248 25674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:23:08 dist-test-slave-d3n8 krb5kdc[25581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 08:23:08.458714 25678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:08.448520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.67:57923 (local address 127.0.0.1:36879)
0619 08:23:08.448812 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:08.448819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:08.448843 (+ 24us) server_negotiation.cc:407] Connection header received
0619 08:23:08.449230 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:08.449238 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:08.449309 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:08.450332 (+ 1023us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:08.450428 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:08.451328 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:08.452103 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:08.452551 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:08.452681 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:08.455258 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:08.455262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:08.455263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:08.455275 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:08.457079 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:08.457512 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:08.457515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:08.457515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:08.457622 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:08.457868 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:08.457869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:08.457870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:08.458114 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:08.458304 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:08.458388 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 08:23:08.458490 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":67,"threads_started":1}
I20260619 08:23:08.459599 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:23:08.467229 25678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:08.460958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.67:49165 (local address 127.0.0.1:36879)
0619 08:23:08.460995 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:08.460996 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:08.461122 (+ 126us) server_negotiation.cc:407] Connection header received
0619 08:23:08.461234 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:08.461235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:08.461246 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:08.461278 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:08.461878 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:08.462419 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:08.463132 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:08.463363 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:08.463451 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:08.464625 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:08.464627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:08.464627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:08.464643 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:08.466051 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:08.466427 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:08.466428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:08.466429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:08.466482 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:08.466716 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:08.466717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:08.466717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:08.466831 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:08.466940 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:08.467062 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 08:23:08.467122 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:23:08.469465 25674 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:23:08.470041 25674 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:23:08.479907 25674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 873b4fae16c14b19824e2c420e4b4a3d: Generated new TSK 0
I20260619 08:23:08.480124 25674 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:23:08.483479 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 25594
W20260619 08:23:08.487813 25579 connection.cc:535] client connection to 127.0.0.1:36879 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36879): Connection reset by peer (error 104)
2026-06-19T08:23:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:23:08 dist-test-slave-d3n8 krb5kdc[25693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:23:08 dist-test-slave-d3n8 krb5kdc[25693](info): set up 2 sockets
Jun 19 08:23:08 dist-test-slave-d3n8 krb5kdc[25693](info): commencing operation
krb5kdc: starting...
W20260619 08:23:09.569657 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:23:09 dist-test-slave-d3n8 krb5kdc[25693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:23:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:23:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 08:23:09.692936 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46363
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:23:09.774507 25706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:23:09.774727 25706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:23:09.774751 25706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:23:09.775928 25706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:23:09.775971 25706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:23:09.775985 25706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:23:09.775998 25706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:23:09.776010 25706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:23:09.777571 25706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:23:09.777828 25706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:23:09.778062 25706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:23:09.782963 25714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:09.783010 25711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:09.782963 25712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:09.783330 25706 server_base.cc:1034] running on GCE node
I20260619 08:23:09.783565 25706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:23:09.783855 25706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:23:09.785030 25706 hybrid_clock.cc:648] HybridClock initialized: now 1781857389785016 us; error 25 us; skew 500 ppm
Jun 19 08:23:09 dist-test-slave-d3n8 krb5kdc[25693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:23:09.787760 25706 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 08:23:09.788249 25706 webserver.cc:460] Webserver started at http://127.0.0.1:34269/ using document root <none> and password file <none>
I20260619 08:23:09.788493 25706 fs_manager.cc:359] Metadata directory not provided
I20260619 08:23:09.788560 25706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:23:09.788748 25706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:23:09.789877 25706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "b9421c8f1977442aae034ea4b1e6d9af"
format_stamp: "Formatted at 2026-06-19 08:23:09 on dist-test-slave-d3n8"
server_key: "2a50c18693c6a00aa401883c044ed818"
server_key_iv: "a7fac170425312bc1a9632f518fe6733"
server_key_version: "encryptionkey@0"
I20260619 08:23:09.790232 25706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "b9421c8f1977442aae034ea4b1e6d9af"
format_stamp: "Formatted at 2026-06-19 08:23:09 on dist-test-slave-d3n8"
server_key: "2a50c18693c6a00aa401883c044ed818"
server_key_iv: "a7fac170425312bc1a9632f518fe6733"
server_key_version: "encryptionkey@0"
I20260619 08:23:09.791590 25706 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:23:09.792480 25721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:23:09.792732 25706 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:23:09.792845 25706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "b9421c8f1977442aae034ea4b1e6d9af"
format_stamp: "Formatted at 2026-06-19 08:23:09 on dist-test-slave-d3n8"
server_key: "2a50c18693c6a00aa401883c044ed818"
server_key_iv: "a7fac170425312bc1a9632f518fe6733"
server_key_version: "encryptionkey@0"
I20260619 08:23:09.792920 25706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:23:09.807518 25706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:23:09.809839 25706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:23:09.810007 25706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:23:09.814527 25706 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46363
I20260619 08:23:09.814584 25773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46363 every 8 connection(s)
I20260619 08:23:09.814976 25706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:23:09.815958 25774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:23:09.817906 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 25706
I20260619 08:23:09.818027 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:23:09.818231 4702 external_mini_cluster.cc:1428] Setting key 007aebacb9ec8a208e2ba2162e64f232
I20260619 08:23:09.818819 25774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9421c8f1977442aae034ea4b1e6d9af: Bootstrap starting.
I20260619 08:23:09.819466 25774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9421c8f1977442aae034ea4b1e6d9af: Neither blocks nor log segments found. Creating new log.
I20260619 08:23:09.819777 25774 log.cc:824] T 00000000000000000000000000000000 P b9421c8f1977442aae034ea4b1e6d9af: Log is configured to *not* fsync() on all Append() calls
I20260619 08:23:09.820483 25774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9421c8f1977442aae034ea4b1e6d9af: No bootstrap required, opened a new log
I20260619 08:23:09.822343 25774 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9421c8f1977442aae034ea4b1e6d9af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9421c8f1977442aae034ea4b1e6d9af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46363 } }
I20260619 08:23:09.822537 25774 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9421c8f1977442aae034ea4b1e6d9af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:23:09.822579 25774 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9421c8f1977442aae034ea4b1e6d9af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9421c8f1977442aae034ea4b1e6d9af, State: Initialized, Role: FOLLOWER
I20260619 08:23:09.822713 25774 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9421c8f1977442aae034ea4b1e6d9af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9421c8f1977442aae034ea4b1e6d9af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46363 } }
I20260619 08:23:09.822803 25774 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9421c8f1977442aae034ea4b1e6d9af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:23:09.822841 25774 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9421c8f1977442aae034ea4b1e6d9af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:23:09.822892 25774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9421c8f1977442aae034ea4b1e6d9af [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:23:09.823529 25774 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9421c8f1977442aae034ea4b1e6d9af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9421c8f1977442aae034ea4b1e6d9af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46363 } }
I20260619 08:23:09.823652 25774 leader_election.cc:302] T 00000000000000000000000000000000 P b9421c8f1977442aae034ea4b1e6d9af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9421c8f1977442aae034ea4b1e6d9af; no voters:
I20260619 08:23:09.823714 25777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:09.818773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.67:40985 (local address 127.0.0.1:46363)
0619 08:23:09.818943 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:09.818951 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:09.818976 (+ 25us) server_negotiation.cc:407] Connection header received
0619 08:23:09.819417 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:09.819423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:09.819469 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:09.819618 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:09.819822 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:09.820829 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:09.821711 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:09.822119 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:09.822241 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:09.823046 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:09.823050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:09.823051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:23:09.823246 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:09.823278 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:23:09.823290 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:09.823386 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:09.823469 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 08:23:09.823496 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260619 08:23:09.823854 25774 leader_election.cc:288] T 00000000000000000000000000000000 P b9421c8f1977442aae034ea4b1e6d9af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:23:09.823886 25779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9421c8f1977442aae034ea4b1e6d9af [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:23:09.824054 25779 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9421c8f1977442aae034ea4b1e6d9af [term 1 LEADER]: Becoming Leader. State: Replica: b9421c8f1977442aae034ea4b1e6d9af, State: Running, Role: LEADER
I20260619 08:23:09.824174 25774 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9421c8f1977442aae034ea4b1e6d9af [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:23:09.824206 25779 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9421c8f1977442aae034ea4b1e6d9af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9421c8f1977442aae034ea4b1e6d9af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46363 } }
I20260619 08:23:09.824726 25780 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9421c8f1977442aae034ea4b1e6d9af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9421c8f1977442aae034ea4b1e6d9af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9421c8f1977442aae034ea4b1e6d9af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46363 } } }
I20260619 08:23:09.824801 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:23:09.824952 25780 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9421c8f1977442aae034ea4b1e6d9af [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:09.824882 25781 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9421c8f1977442aae034ea4b1e6d9af [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9421c8f1977442aae034ea4b1e6d9af. Latest consensus state: current_term: 1 leader_uuid: "b9421c8f1977442aae034ea4b1e6d9af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9421c8f1977442aae034ea4b1e6d9af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46363 } } }
I20260619 08:23:09.825093 25781 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9421c8f1977442aae034ea4b1e6d9af [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:09.825443 25788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:23:09.825948 25788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:23:09.827683 25788 catalog_manager.cc:1347] Generated new cluster ID: 3670e18064154fcb87623106211e74b1
I20260619 08:23:09.827749 25788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:23:09.830381 25777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:09.826511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.67:34639 (local address 127.0.0.1:46363)
0619 08:23:09.826548 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:09.826550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:09.826686 (+ 136us) server_negotiation.cc:407] Connection header received
0619 08:23:09.826737 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:09.826738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:09.826748 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:09.826814 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:09.826967 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:09.827715 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:09.828792 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:09.829123 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:09.829185 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:09.830043 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:09.830045 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:09.830045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:23:09.830050 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:09.830065 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:23:09.830070 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:09.830120 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:09.830283 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 08:23:09.830300 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:23:09.842278 25788 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:23:09.842859 25788 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:23:09.852705 25788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9421c8f1977442aae034ea4b1e6d9af: Generated new TSK 0
I20260619 08:23:09.852969 25788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:23:09.864591 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 25706
W20260619 08:23:09.868995 25691 connection.cc:535] client connection to 127.0.0.1:46363 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46363): Connection reset by peer (error 104)
2026-06-19T08:23:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:23:09 dist-test-slave-d3n8 krb5kdc[25805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:23:09 dist-test-slave-d3n8 krb5kdc[25805](info): set up 2 sockets
Jun 19 08:23:09 dist-test-slave-d3n8 krb5kdc[25805](info): commencing operation
krb5kdc: starting...
W20260619 08:23:10.909677 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:23:10 dist-test-slave-d3n8 krb5kdc[25805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:23:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:23:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 08:23:11.033036 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43047
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:23:11.113713 25818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:23:11.113924 25818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:23:11.113947 25818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:23:11.115154 25818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:23:11.115203 25818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:23:11.115218 25818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:23:11.115239 25818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:23:11.115252 25818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:23:11.116923 25818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:23:11.117193 25818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:23:11.117523 25818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:23:11.122581 25824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:11.122591 25823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:11.122779 25826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:11.123339 25818 server_base.cc:1034] running on GCE node
I20260619 08:23:11.123591 25818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:23:11.123850 25818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:23:11.124996 25818 hybrid_clock.cc:648] HybridClock initialized: now 1781857391124976 us; error 36 us; skew 500 ppm
Jun 19 08:23:11 dist-test-slave-d3n8 krb5kdc[25805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:23:11.127764 25818 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 08:23:11.128149 25818 webserver.cc:460] Webserver started at http://127.0.0.1:41759/ using document root <none> and password file <none>
I20260619 08:23:11.128397 25818 fs_manager.cc:359] Metadata directory not provided
I20260619 08:23:11.128470 25818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:23:11.128623 25818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:23:11.129774 25818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "c1480c1f18db4c73903975e409eef975"
format_stamp: "Formatted at 2026-06-19 08:23:11 on dist-test-slave-d3n8"
server_key: "4ccd1ac63b4e7c749449164a64ed348d"
server_key_iv: "5b20a79f0d30d8aea756d18e181a1447"
server_key_version: "encryptionkey@0"
I20260619 08:23:11.130138 25818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "c1480c1f18db4c73903975e409eef975"
format_stamp: "Formatted at 2026-06-19 08:23:11 on dist-test-slave-d3n8"
server_key: "4ccd1ac63b4e7c749449164a64ed348d"
server_key_iv: "5b20a79f0d30d8aea756d18e181a1447"
server_key_version: "encryptionkey@0"
I20260619 08:23:11.131510 25818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:23:11.132261 25833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:23:11.132477 25818 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:23:11.132596 25818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "c1480c1f18db4c73903975e409eef975"
format_stamp: "Formatted at 2026-06-19 08:23:11 on dist-test-slave-d3n8"
server_key: "4ccd1ac63b4e7c749449164a64ed348d"
server_key_iv: "5b20a79f0d30d8aea756d18e181a1447"
server_key_version: "encryptionkey@0"
I20260619 08:23:11.132678 25818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:23:11.136089 25818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:23:11.138528 25818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:23:11.138687 25818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:23:11.142565 25818 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43047
I20260619 08:23:11.142597 25885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43047 every 8 connection(s)
I20260619 08:23:11.142925 25818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:23:11.143882 25886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:23:11.146239 25886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1480c1f18db4c73903975e409eef975: Bootstrap starting.
I20260619 08:23:11.146816 25886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1480c1f18db4c73903975e409eef975: Neither blocks nor log segments found. Creating new log.
I20260619 08:23:11.147099 25886 log.cc:824] T 00000000000000000000000000000000 P c1480c1f18db4c73903975e409eef975: Log is configured to *not* fsync() on all Append() calls
I20260619 08:23:11.147789 25886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1480c1f18db4c73903975e409eef975: No bootstrap required, opened a new log
I20260619 08:23:11.148020 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 25818
I20260619 08:23:11.148156 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:23:11.148437 4702 external_mini_cluster.cc:1428] Setting key 66e730ec1164565ebe633c604ec71ea7
I20260619 08:23:11.150279 25886 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1480c1f18db4c73903975e409eef975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1480c1f18db4c73903975e409eef975" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43047 } }
I20260619 08:23:11.150427 25886 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1480c1f18db4c73903975e409eef975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:23:11.150482 25886 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1480c1f18db4c73903975e409eef975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1480c1f18db4c73903975e409eef975, State: Initialized, Role: FOLLOWER
I20260619 08:23:11.150607 25886 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1480c1f18db4c73903975e409eef975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1480c1f18db4c73903975e409eef975" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43047 } }
I20260619 08:23:11.150701 25886 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1480c1f18db4c73903975e409eef975 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:23:11.150735 25886 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1480c1f18db4c73903975e409eef975 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:23:11.150779 25886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1480c1f18db4c73903975e409eef975 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:23:11.151185 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:11.148985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.67:44153 (local address 127.0.0.1:43047)
0619 08:23:11.149217 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:11.149227 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:11.149265 (+ 38us) server_negotiation.cc:407] Connection header received
0619 08:23:11.149817 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:11.149826 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:11.149859 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:11.150010 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:11.150698 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:11.150701 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:11.150702 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:23:11.150707 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:11.150765 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:23:11.150780 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:11.150854 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:11.150933 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 08:23:11.150975 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":100,"threads_started":1}
I20260619 08:23:11.151906 25886 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1480c1f18db4c73903975e409eef975 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1480c1f18db4c73903975e409eef975" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43047 } }
I20260619 08:23:11.152015 25886 leader_election.cc:302] T 00000000000000000000000000000000 P c1480c1f18db4c73903975e409eef975 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1480c1f18db4c73903975e409eef975; no voters:
I20260619 08:23:11.152189 25886 leader_election.cc:288] T 00000000000000000000000000000000 P c1480c1f18db4c73903975e409eef975 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:23:11.152199 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:23:11.152387 25892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1480c1f18db4c73903975e409eef975 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:23:11.152460 25886 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1480c1f18db4c73903975e409eef975 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:23:11.152597 25892 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1480c1f18db4c73903975e409eef975 [term 1 LEADER]: Becoming Leader. State: Replica: c1480c1f18db4c73903975e409eef975, State: Running, Role: LEADER
I20260619 08:23:11.152769 25892 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1480c1f18db4c73903975e409eef975 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1480c1f18db4c73903975e409eef975" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43047 } }
I20260619 08:23:11.153164 25892 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1480c1f18db4c73903975e409eef975 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1480c1f18db4c73903975e409eef975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1480c1f18db4c73903975e409eef975" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43047 } } }
I20260619 08:23:11.153283 25892 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1480c1f18db4c73903975e409eef975 [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:11.153473 25892 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1480c1f18db4c73903975e409eef975 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1480c1f18db4c73903975e409eef975. Latest consensus state: current_term: 1 leader_uuid: "c1480c1f18db4c73903975e409eef975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1480c1f18db4c73903975e409eef975" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43047 } } }
I20260619 08:23:11.153604 25892 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1480c1f18db4c73903975e409eef975 [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:11.153599 25902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:23:11.154238 25902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:23:11.155812 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:11.154539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.67:35205 (local address 127.0.0.1:43047)
0619 08:23:11.154594 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:11.154595 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:11.154717 (+ 122us) server_negotiation.cc:407] Connection header received
0619 08:23:11.154790 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:11.154791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:11.154812 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:11.154864 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:11.155578 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:11.155579 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:11.155579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:23:11.155584 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:11.155598 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:23:11.155610 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:11.155675 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:11.155742 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 08:23:11.155755 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 08:23:11.156136 25902 catalog_manager.cc:1347] Generated new cluster ID: bddaf1a638474389a994b962753d78d2
I20260619 08:23:11.156215 25902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:23:11.163931 25902 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:23:11.164086 25902 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:23:11.172894 25902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1480c1f18db4c73903975e409eef975: Generated new TSK 0
I20260619 08:23:11.173134 25902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:23:11.175741 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 25818
2026-06-19T08:23:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:23:11 dist-test-slave-d3n8 krb5kdc[25917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:23:11 dist-test-slave-d3n8 krb5kdc[25917](info): set up 2 sockets
Jun 19 08:23:11 dist-test-slave-d3n8 krb5kdc[25917](info): commencing operation
krb5kdc: starting...
W20260619 08:23:12.230574 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:23:12 dist-test-slave-d3n8 krb5kdc[25917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:23:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:23:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 08:23:12.357055 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39957
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:23:12.446615 25930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:23:12.446857 25930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:23:12.446883 25930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:23:12.448294 25930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:23:12.448349 25930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:23:12.448366 25930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:23:12.448380 25930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:23:12.448395 25930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:23:12.450268 25930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39957
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:23:12.450563 25930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:23:12.450829 25930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:23:12.456314 25936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:12.456394 25938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:12.456291 25935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:12.456521 25930 server_base.cc:1034] running on GCE node
I20260619 08:23:12.456770 25930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:23:12.457095 25930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:23:12.458245 25930 hybrid_clock.cc:648] HybridClock initialized: now 1781857392458212 us; error 48 us; skew 500 ppm
Jun 19 08:23:12 dist-test-slave-d3n8 krb5kdc[25917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:23:12.466071 25930 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 08:23:12.466638 25930 webserver.cc:460] Webserver started at http://127.0.0.1:46633/ using document root <none> and password file <none>
I20260619 08:23:12.466919 25930 fs_manager.cc:359] Metadata directory not provided
I20260619 08:23:12.466969 25930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:23:12.467108 25930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:23:12.468230 25930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "30be049edb794e0f97b0d2ebb2af945e"
format_stamp: "Formatted at 2026-06-19 08:23:12 on dist-test-slave-d3n8"
server_key: "2e643d815b1ac5fc570d499c023f01c5"
server_key_iv: "82c2884468c99afc62dfcbc7f9b91131"
server_key_version: "encryptionkey@0"
I20260619 08:23:12.468600 25930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "30be049edb794e0f97b0d2ebb2af945e"
format_stamp: "Formatted at 2026-06-19 08:23:12 on dist-test-slave-d3n8"
server_key: "2e643d815b1ac5fc570d499c023f01c5"
server_key_iv: "82c2884468c99afc62dfcbc7f9b91131"
server_key_version: "encryptionkey@0"
I20260619 08:23:12.470134 25930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:23:12.470990 25945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:23:12.471361 25930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:23:12.471458 25930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "30be049edb794e0f97b0d2ebb2af945e"
format_stamp: "Formatted at 2026-06-19 08:23:12 on dist-test-slave-d3n8"
server_key: "2e643d815b1ac5fc570d499c023f01c5"
server_key_iv: "82c2884468c99afc62dfcbc7f9b91131"
server_key_version: "encryptionkey@0"
I20260619 08:23:12.471561 25930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:23:12.490054 25930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:23:12.492118 25930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:23:12.492298 25930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:23:12.496498 25930 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39957
I20260619 08:23:12.496538 25997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39957 every 8 connection(s)
I20260619 08:23:12.496927 25930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:23:12.498031 25998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:23:12.500386 25998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30be049edb794e0f97b0d2ebb2af945e: Bootstrap starting.
I20260619 08:23:12.500946 25998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30be049edb794e0f97b0d2ebb2af945e: Neither blocks nor log segments found. Creating new log.
I20260619 08:23:12.501192 25998 log.cc:824] T 00000000000000000000000000000000 P 30be049edb794e0f97b0d2ebb2af945e: Log is configured to *not* fsync() on all Append() calls
I20260619 08:23:12.501962 25998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30be049edb794e0f97b0d2ebb2af945e: No bootstrap required, opened a new log
I20260619 08:23:12.502779 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 25930
I20260619 08:23:12.502935 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:23:12.503182 4702 external_mini_cluster.cc:1428] Setting key 044e17ab7130efd67d2763b628152bef
I20260619 08:23:12.505059 25998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30be049edb794e0f97b0d2ebb2af945e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30be049edb794e0f97b0d2ebb2af945e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39957 } }
I20260619 08:23:12.505245 25998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30be049edb794e0f97b0d2ebb2af945e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:23:12.505286 25998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30be049edb794e0f97b0d2ebb2af945e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30be049edb794e0f97b0d2ebb2af945e, State: Initialized, Role: FOLLOWER
I20260619 08:23:12.505470 25998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30be049edb794e0f97b0d2ebb2af945e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30be049edb794e0f97b0d2ebb2af945e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39957 } }
I20260619 08:23:12.505537 25998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30be049edb794e0f97b0d2ebb2af945e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:23:12.505561 25998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30be049edb794e0f97b0d2ebb2af945e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:23:12.505625 25998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30be049edb794e0f97b0d2ebb2af945e [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:23:12.506711 25998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30be049edb794e0f97b0d2ebb2af945e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30be049edb794e0f97b0d2ebb2af945e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39957 } }
I20260619 08:23:12.506819 25998 leader_election.cc:302] T 00000000000000000000000000000000 P 30be049edb794e0f97b0d2ebb2af945e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30be049edb794e0f97b0d2ebb2af945e; no voters:
I20260619 08:23:12.506969 25998 leader_election.cc:288] T 00000000000000000000000000000000 P 30be049edb794e0f97b0d2ebb2af945e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:23:12.507055 26003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30be049edb794e0f97b0d2ebb2af945e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:23:12.507205 25998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30be049edb794e0f97b0d2ebb2af945e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:23:12.507331 26003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30be049edb794e0f97b0d2ebb2af945e [term 1 LEADER]: Becoming Leader. State: Replica: 30be049edb794e0f97b0d2ebb2af945e, State: Running, Role: LEADER
I20260619 08:23:12.507465 26003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30be049edb794e0f97b0d2ebb2af945e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30be049edb794e0f97b0d2ebb2af945e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39957 } }
I20260619 08:23:12.507882 26004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30be049edb794e0f97b0d2ebb2af945e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30be049edb794e0f97b0d2ebb2af945e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30be049edb794e0f97b0d2ebb2af945e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39957 } } }
I20260619 08:23:12.507961 26004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30be049edb794e0f97b0d2ebb2af945e [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:12.508244 26005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30be049edb794e0f97b0d2ebb2af945e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30be049edb794e0f97b0d2ebb2af945e. Latest consensus state: current_term: 1 leader_uuid: "30be049edb794e0f97b0d2ebb2af945e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30be049edb794e0f97b0d2ebb2af945e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39957 } } }
I20260619 08:23:12.508316 26005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30be049edb794e0f97b0d2ebb2af945e [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:12.508589 26012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:23:12.509121 26012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:23:12.511020 26012 catalog_manager.cc:1347] Generated new cluster ID: c575c2fac217409cb9748f25ae4d5b69
I20260619 08:23:12.511073 26012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:23:12 dist-test-slave-d3n8 krb5kdc[25917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 08:23:12.516223 26002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:12.503773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.67:42925 (local address 127.0.0.1:39957)
0619 08:23:12.503998 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:12.504007 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:12.504044 (+ 37us) server_negotiation.cc:407] Connection header received
0619 08:23:12.505012 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:12.505022 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:12.505090 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:12.505246 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:12.506139 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:12.507579 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:12.508766 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:12.509443 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:12.509578 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:12.512450 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:12.512454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:12.512455 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:12.512467 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:12.514466 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:12.514903 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:12.514906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:12.514906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:12.515042 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:12.515273 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:12.515273 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:12.515274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:12.515602 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:12.515796 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:12.515875 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 08:23:12.515976 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":108,"threads_started":1}
I20260619 08:23:12.517401 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:23:12.525454 26012 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:23:12.525918 26012 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:23:12.530447 26002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:12.518971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.67:57171 (local address 127.0.0.1:39957)
0619 08:23:12.519006 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:12.519007 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:12.519015 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:23:12.519086 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:12.519086 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:12.519096 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:12.519144 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:12.519791 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:12.520527 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:12.521602 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:12.521933 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:12.521993 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:12.523417 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:12.523419 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:12.523419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:12.523426 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:12.525014 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:12.529551 (+ 4537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:12.529552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:12.529553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:12.529627 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:12.529883 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:12.529884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:12.529884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:12.530030 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:12.530167 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:12.530285 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 08:23:12.530355 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:23:12.535377 26012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30be049edb794e0f97b0d2ebb2af945e: Generated new TSK 0
I20260619 08:23:12.535562 26012 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:23:12.540547 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 25930
W20260619 08:23:12.545112 25915 connection.cc:535] client connection to 127.0.0.1:39957 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39957): Connection reset by peer (error 104)
2026-06-19T08:23:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:23:12 dist-test-slave-d3n8 krb5kdc[26029](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:23:12 dist-test-slave-d3n8 krb5kdc[26029](info): set up 2 sockets
Jun 19 08:23:12 dist-test-slave-d3n8 krb5kdc[26029](info): commencing operation
krb5kdc: starting...
W20260619 08:23:13.581648 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:23:13 dist-test-slave-d3n8 krb5kdc[26029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:23:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:23:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 08:23:13.705813 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38031
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:23:13.792272 26042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:23:13.792497 26042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:23:13.792523 26042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:23:13.793689 26042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:23:13.793728 26042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:23:13.793743 26042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:23:13.793782 26042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:23:13.793795 26042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:23:13.795272 26042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38031
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:23:13.795485 26042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:23:13.795673 26042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:23:13.800987 26048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:13.800979 26047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:13.800998 26042 server_base.cc:1034] running on GCE node
W20260619 08:23:13.800979 26050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:13.801517 26042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:23:13.801862 26042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:23:13.803050 26042 hybrid_clock.cc:648] HybridClock initialized: now 1781857393803032 us; error 40 us; skew 500 ppm
Jun 19 08:23:13 dist-test-slave-d3n8 krb5kdc[26029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:23:13.806346 26042 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 08:23:13.806895 26042 webserver.cc:460] Webserver started at http://127.0.0.1:39029/ using document root <none> and password file <none>
I20260619 08:23:13.807142 26042 fs_manager.cc:359] Metadata directory not provided
I20260619 08:23:13.807215 26042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:23:13.807369 26042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:23:13.808507 26042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "346327e51fdd474a844e2f7ec7827f93"
format_stamp: "Formatted at 2026-06-19 08:23:13 on dist-test-slave-d3n8"
server_key: "fef17130052b5e093d46cf821912da7e"
server_key_iv: "fceabfcbc6a1cd855bb936ef8194a90b"
server_key_version: "encryptionkey@0"
I20260619 08:23:13.808887 26042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "346327e51fdd474a844e2f7ec7827f93"
format_stamp: "Formatted at 2026-06-19 08:23:13 on dist-test-slave-d3n8"
server_key: "fef17130052b5e093d46cf821912da7e"
server_key_iv: "fceabfcbc6a1cd855bb936ef8194a90b"
server_key_version: "encryptionkey@0"
I20260619 08:23:13.810545 26042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:23:13.811623 26057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:23:13.811862 26042 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:23:13.812004 26042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "346327e51fdd474a844e2f7ec7827f93"
format_stamp: "Formatted at 2026-06-19 08:23:13 on dist-test-slave-d3n8"
server_key: "fef17130052b5e093d46cf821912da7e"
server_key_iv: "fceabfcbc6a1cd855bb936ef8194a90b"
server_key_version: "encryptionkey@0"
I20260619 08:23:13.812106 26042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:23:13.827018 26042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:23:13.829602 26042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:23:13.829782 26042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:23:13.834060 26042 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38031
I20260619 08:23:13.834200 26109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38031 every 8 connection(s)
I20260619 08:23:13.834538 26042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:23:13.835554 26110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:23:13.838014 26110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 346327e51fdd474a844e2f7ec7827f93: Bootstrap starting.
I20260619 08:23:13.838568 26110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 346327e51fdd474a844e2f7ec7827f93: Neither blocks nor log segments found. Creating new log.
I20260619 08:23:13.838810 26110 log.cc:824] T 00000000000000000000000000000000 P 346327e51fdd474a844e2f7ec7827f93: Log is configured to *not* fsync() on all Append() calls
I20260619 08:23:13.839561 26110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 346327e51fdd474a844e2f7ec7827f93: No bootstrap required, opened a new log
I20260619 08:23:13.841231 26110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 346327e51fdd474a844e2f7ec7827f93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "346327e51fdd474a844e2f7ec7827f93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38031 } }
I20260619 08:23:13.841454 26110 raft_consensus.cc:374] T 00000000000000000000000000000000 P 346327e51fdd474a844e2f7ec7827f93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:23:13.841526 26110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 346327e51fdd474a844e2f7ec7827f93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346327e51fdd474a844e2f7ec7827f93, State: Initialized, Role: FOLLOWER
I20260619 08:23:13.841688 26110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 346327e51fdd474a844e2f7ec7827f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "346327e51fdd474a844e2f7ec7827f93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38031 } }
I20260619 08:23:13.841782 26110 raft_consensus.cc:388] T 00000000000000000000000000000000 P 346327e51fdd474a844e2f7ec7827f93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:23:13.841807 26110 raft_consensus.cc:482] T 00000000000000000000000000000000 P 346327e51fdd474a844e2f7ec7827f93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:23:13.841836 26110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 346327e51fdd474a844e2f7ec7827f93 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:23:13.842043 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 26042
I20260619 08:23:13.842165 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:23:13.842360 4702 external_mini_cluster.cc:1428] Setting key d4db5b1a2f017423176ce5a83338f054
I20260619 08:23:13.842644 26110 raft_consensus.cc:504] T 00000000000000000000000000000000 P 346327e51fdd474a844e2f7ec7827f93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "346327e51fdd474a844e2f7ec7827f93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38031 } }
I20260619 08:23:13.842772 26110 leader_election.cc:302] T 00000000000000000000000000000000 P 346327e51fdd474a844e2f7ec7827f93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 346327e51fdd474a844e2f7ec7827f93; no voters:
I20260619 08:23:13.843025 26110 leader_election.cc:288] T 00000000000000000000000000000000 P 346327e51fdd474a844e2f7ec7827f93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:23:13.843082 26114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 346327e51fdd474a844e2f7ec7827f93 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:23:13.843276 26114 raft_consensus.cc:686] T 00000000000000000000000000000000 P 346327e51fdd474a844e2f7ec7827f93 [term 1 LEADER]: Becoming Leader. State: Replica: 346327e51fdd474a844e2f7ec7827f93, State: Running, Role: LEADER
I20260619 08:23:13.843382 26110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 346327e51fdd474a844e2f7ec7827f93 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:23:13.843439 26114 consensus_queue.cc:237] T 00000000000000000000000000000000 P 346327e51fdd474a844e2f7ec7827f93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "346327e51fdd474a844e2f7ec7827f93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38031 } }
I20260619 08:23:13.843786 26114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 346327e51fdd474a844e2f7ec7827f93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 346327e51fdd474a844e2f7ec7827f93. Latest consensus state: current_term: 1 leader_uuid: "346327e51fdd474a844e2f7ec7827f93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "346327e51fdd474a844e2f7ec7827f93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38031 } } }
I20260619 08:23:13.843945 26114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 346327e51fdd474a844e2f7ec7827f93 [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:13.844249 26124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:23:13.844899 26124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:23:13.845033 26116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 346327e51fdd474a844e2f7ec7827f93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "346327e51fdd474a844e2f7ec7827f93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "346327e51fdd474a844e2f7ec7827f93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38031 } } }
I20260619 08:23:13.845323 26116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 346327e51fdd474a844e2f7ec7827f93 [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:13.846822 26124 catalog_manager.cc:1347] Generated new cluster ID: 0ede289a94374c0f94bb1167fb4915ec
I20260619 08:23:13.846880 26124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:23:13.850065 26115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:13.842914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.67:36385 (local address 127.0.0.1:38031)
0619 08:23:13.843335 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:13.843345 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:13.843378 (+ 33us) server_negotiation.cc:407] Connection header received
0619 08:23:13.843501 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:13.843508 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:13.843569 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:13.843725 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:13.844045 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:13.845746 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:13.847059 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:13.847759 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:13.847861 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:13.848893 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:13.848897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:13.848898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:23:13.849129 (+ 231us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:13.849169 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:23:13.849186 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:13.849264 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:13.849783 (+ 519us) server_negotiation.cc:299] Negotiation successful
0619 08:23:13.849816 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":119,"threads_started":1}
I20260619 08:23:13.851172 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:23:13.856477 26115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:13.852679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.67:44871 (local address 127.0.0.1:38031)
0619 08:23:13.852723 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:13.852725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:13.852903 (+ 178us) server_negotiation.cc:407] Connection header received
0619 08:23:13.852985 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:13.852986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:13.852996 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:13.853024 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:13.853228 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:13.854006 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:13.855045 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:13.855364 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:13.855399 (+ 35us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:13.856215 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:13.856217 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:13.856217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:23:13.856222 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:13.856238 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:23:13.856242 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:13.856271 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:13.856378 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 08:23:13.856392 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:23:13.862632 26124 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:23:13.863287 26124 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:23:13.869550 26124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 346327e51fdd474a844e2f7ec7827f93: Generated new TSK 0
I20260619 08:23:13.869761 26124 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:23:13.873723 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 26042
W20260619 08:23:13.878276 26027 connection.cc:535] client connection to 127.0.0.1:38031 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38031): Connection reset by peer (error 104)
2026-06-19T08:23:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:23:13 dist-test-slave-d3n8 krb5kdc[26141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:23:13 dist-test-slave-d3n8 krb5kdc[26141](info): set up 2 sockets
Jun 19 08:23:13 dist-test-slave-d3n8 krb5kdc[26141](info): commencing operation
krb5kdc: starting...
W20260619 08:23:14.929617 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:23:15 dist-test-slave-d3n8 krb5kdc[26141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:23:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:23:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 08:23:15.058615 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41149
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:23:15.139971 26154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:23:15.140187 26154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:23:15.140211 26154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:23:15.141427 26154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:23:15.141474 26154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:23:15.141487 26154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:23:15.141500 26154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:23:15.141512 26154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:23:15.143007 26154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41149
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:23:15.143265 26154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:23:15.143477 26154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:23:15.148610 26160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:15.148617 26159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:15.148649 26154 server_base.cc:1034] running on GCE node
W20260619 08:23:15.148625 26162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:15.149152 26154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:23:15.149514 26154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:23:15.150695 26154 hybrid_clock.cc:648] HybridClock initialized: now 1781857395150689 us; error 38 us; skew 500 ppm
Jun 19 08:23:15 dist-test-slave-d3n8 krb5kdc[26141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:23:15.153501 26154 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 08:23:15.153951 26154 webserver.cc:460] Webserver started at http://127.0.0.1:45133/ using document root <none> and password file <none>
I20260619 08:23:15.154209 26154 fs_manager.cc:359] Metadata directory not provided
I20260619 08:23:15.154284 26154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:23:15.154431 26154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:23:15.155491 26154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "ba78e083e4cb49c2b432f23f245e0a15"
format_stamp: "Formatted at 2026-06-19 08:23:15 on dist-test-slave-d3n8"
server_key: "ea07ee1a82e7a7fcb51ff13e1814cee3"
server_key_iv: "46e638ccda0634ae56781d2974bbe5bb"
server_key_version: "encryptionkey@0"
I20260619 08:23:15.155848 26154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "ba78e083e4cb49c2b432f23f245e0a15"
format_stamp: "Formatted at 2026-06-19 08:23:15 on dist-test-slave-d3n8"
server_key: "ea07ee1a82e7a7fcb51ff13e1814cee3"
server_key_iv: "46e638ccda0634ae56781d2974bbe5bb"
server_key_version: "encryptionkey@0"
I20260619 08:23:15.157294 26154 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:23:15.158110 26169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:23:15.158313 26154 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:23:15.158420 26154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "ba78e083e4cb49c2b432f23f245e0a15"
format_stamp: "Formatted at 2026-06-19 08:23:15 on dist-test-slave-d3n8"
server_key: "ea07ee1a82e7a7fcb51ff13e1814cee3"
server_key_iv: "46e638ccda0634ae56781d2974bbe5bb"
server_key_version: "encryptionkey@0"
I20260619 08:23:15.158499 26154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:23:15.162021 26154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:23:15.164361 26154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:23:15.164536 26154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:23:15.168578 26154 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41149
I20260619 08:23:15.168711 26221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41149 every 8 connection(s)
I20260619 08:23:15.168998 26154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:23:15.170064 26222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:23:15.172344 26222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba78e083e4cb49c2b432f23f245e0a15: Bootstrap starting.
I20260619 08:23:15.172890 26222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba78e083e4cb49c2b432f23f245e0a15: Neither blocks nor log segments found. Creating new log.
I20260619 08:23:15.173120 26222 log.cc:824] T 00000000000000000000000000000000 P ba78e083e4cb49c2b432f23f245e0a15: Log is configured to *not* fsync() on all Append() calls
I20260619 08:23:15.173833 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 26154
I20260619 08:23:15.173872 26222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba78e083e4cb49c2b432f23f245e0a15: No bootstrap required, opened a new log
I20260619 08:23:15.173971 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:23:15.174170 4702 external_mini_cluster.cc:1428] Setting key c02dc430a8cd8dd69f35db14323ee4c9
I20260619 08:23:15.175952 26222 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba78e083e4cb49c2b432f23f245e0a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba78e083e4cb49c2b432f23f245e0a15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41149 } }
I20260619 08:23:15.176157 26222 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba78e083e4cb49c2b432f23f245e0a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:23:15.176204 26222 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba78e083e4cb49c2b432f23f245e0a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba78e083e4cb49c2b432f23f245e0a15, State: Initialized, Role: FOLLOWER
I20260619 08:23:15.176396 26222 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba78e083e4cb49c2b432f23f245e0a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba78e083e4cb49c2b432f23f245e0a15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41149 } }
I20260619 08:23:15.176507 26222 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba78e083e4cb49c2b432f23f245e0a15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:23:15.176584 26222 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba78e083e4cb49c2b432f23f245e0a15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:23:15.176667 26222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba78e083e4cb49c2b432f23f245e0a15 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:23:15.177326 26226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:15.174709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.67:37203 (local address 127.0.0.1:41149)
0619 08:23:15.174887 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:15.174902 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:15.174931 (+ 29us) server_negotiation.cc:407] Connection header received
0619 08:23:15.175921 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:15.175928 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:15.175956 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:15.176079 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:15.176839 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:15.176842 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:15.176843 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:23:15.176847 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:15.176895 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:23:15.176906 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:15.176988 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:15.177054 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 08:23:15.177105 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":102,"threads_started":1}
I20260619 08:23:15.178041 26222 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba78e083e4cb49c2b432f23f245e0a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba78e083e4cb49c2b432f23f245e0a15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41149 } }
I20260619 08:23:15.178187 26222 leader_election.cc:302] T 00000000000000000000000000000000 P ba78e083e4cb49c2b432f23f245e0a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba78e083e4cb49c2b432f23f245e0a15; no voters:
I20260619 08:23:15.178388 26222 leader_election.cc:288] T 00000000000000000000000000000000 P ba78e083e4cb49c2b432f23f245e0a15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:23:15.178421 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:23:15.178418 26227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba78e083e4cb49c2b432f23f245e0a15 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:23:15.178543 26227 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba78e083e4cb49c2b432f23f245e0a15 [term 1 LEADER]: Becoming Leader. State: Replica: ba78e083e4cb49c2b432f23f245e0a15, State: Running, Role: LEADER
I20260619 08:23:15.178627 26227 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba78e083e4cb49c2b432f23f245e0a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba78e083e4cb49c2b432f23f245e0a15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41149 } }
I20260619 08:23:15.178663 26222 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba78e083e4cb49c2b432f23f245e0a15 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:23:15.178916 26228 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba78e083e4cb49c2b432f23f245e0a15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba78e083e4cb49c2b432f23f245e0a15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba78e083e4cb49c2b432f23f245e0a15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41149 } } }
I20260619 08:23:15.178984 26228 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba78e083e4cb49c2b432f23f245e0a15 [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:15.179148 26229 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba78e083e4cb49c2b432f23f245e0a15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba78e083e4cb49c2b432f23f245e0a15. Latest consensus state: current_term: 1 leader_uuid: "ba78e083e4cb49c2b432f23f245e0a15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba78e083e4cb49c2b432f23f245e0a15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41149 } } }
I20260619 08:23:15.179227 26229 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba78e083e4cb49c2b432f23f245e0a15 [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:15.179216 26231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:23:15.179783 26231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:23:15.181457 26226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:15.180261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.67:46801 (local address 127.0.0.1:41149)
0619 08:23:15.180317 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:15.180319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:15.180466 (+ 147us) server_negotiation.cc:407] Connection header received
0619 08:23:15.180537 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:15.180538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:15.180543 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:15.180604 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:15.181189 (+ 585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:15.181191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:15.181191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:23:15.181195 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:15.181210 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:23:15.181214 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:15.181266 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:15.181352 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 08:23:15.181366 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 08:23:15.181690 26231 catalog_manager.cc:1347] Generated new cluster ID: 36b47fa898f9448c9a70b3ac290d3ee0
I20260619 08:23:15.181734 26231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:23:15.200771 26231 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:23:15.200965 26231 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:23:15.207506 26231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba78e083e4cb49c2b432f23f245e0a15: Generated new TSK 0
I20260619 08:23:15.207731 26231 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:23:15.212477 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 26154
2026-06-19T08:23:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:23:15 dist-test-slave-d3n8 krb5kdc[26253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:23:15 dist-test-slave-d3n8 krb5kdc[26253](info): set up 2 sockets
Jun 19 08:23:15 dist-test-slave-d3n8 krb5kdc[26253](info): commencing operation
krb5kdc: starting...
W20260619 08:23:16.253633 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:23:16 dist-test-slave-d3n8 krb5kdc[26253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:23:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:23:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:23:16.381774 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:44609
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:42461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:44609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:23:16.466460 26266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:23:16.466684 26266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:23:16.466710 26266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:23:16.467844 26266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:23:16.467882 26266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:23:16.467897 26266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:23:16.467911 26266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:23:16.467922 26266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:23:16.469462 26266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:42461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:44609
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:44609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:23:16.469700 26266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:23:16.469882 26266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:23:16.475000 26272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:16.474994 26274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:16.474998 26271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:16.475061 26266 server_base.cc:1034] running on GCE node
I20260619 08:23:16.475426 26266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:23:16.475695 26266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:23:16.476877 26266 hybrid_clock.cc:648] HybridClock initialized: now 1781857396476858 us; error 49 us; skew 500 ppm
Jun 19 08:23:16 dist-test-slave-d3n8 krb5kdc[26253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857396, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:23:16.479786 26266 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:23:16.480387 26266 webserver.cc:460] Webserver started at http://127.4.151.190:43859/ using document root <none> and password file <none>
I20260619 08:23:16.480680 26266 fs_manager.cc:359] Metadata directory not provided
I20260619 08:23:16.480758 26266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:23:16.480923 26266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:23:16.482191 26266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "ad3cad486ceb4d4abba0c261c09af0ca"
format_stamp: "Formatted at 2026-06-19 08:23:16 on dist-test-slave-d3n8"
server_key: "131f10a680830d00ba96955d60b339b9"
server_key_iv: "6718930bae7328f81076b7b328d1b7ef"
server_key_version: "encryptionkey@0"
I20260619 08:23:16.482581 26266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "ad3cad486ceb4d4abba0c261c09af0ca"
format_stamp: "Formatted at 2026-06-19 08:23:16 on dist-test-slave-d3n8"
server_key: "131f10a680830d00ba96955d60b339b9"
server_key_iv: "6718930bae7328f81076b7b328d1b7ef"
server_key_version: "encryptionkey@0"
I20260619 08:23:16.484108 26266 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 08:23:16.485073 26281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:23:16.485308 26266 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:23:16.485441 26266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "ad3cad486ceb4d4abba0c261c09af0ca"
format_stamp: "Formatted at 2026-06-19 08:23:16 on dist-test-slave-d3n8"
server_key: "131f10a680830d00ba96955d60b339b9"
server_key_iv: "6718930bae7328f81076b7b328d1b7ef"
server_key_version: "encryptionkey@0"
I20260619 08:23:16.485538 26266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:23:16.510215 26266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:23:16.512920 26266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:23:16.513110 26266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:23:16.517531 26266 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:44609
I20260619 08:23:16.517614 26333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:44609 every 8 connection(s)
I20260619 08:23:16.518020 26266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:23:16.518978 26334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:23:16.521508 26334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad3cad486ceb4d4abba0c261c09af0ca: Bootstrap starting.
I20260619 08:23:16.522085 26334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad3cad486ceb4d4abba0c261c09af0ca: Neither blocks nor log segments found. Creating new log.
I20260619 08:23:16.522351 26334 log.cc:824] T 00000000000000000000000000000000 P ad3cad486ceb4d4abba0c261c09af0ca: Log is configured to *not* fsync() on all Append() calls
I20260619 08:23:16.523093 26334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad3cad486ceb4d4abba0c261c09af0ca: No bootstrap required, opened a new log
I20260619 08:23:16.524786 26334 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad3cad486ceb4d4abba0c261c09af0ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3cad486ceb4d4abba0c261c09af0ca" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44609 } }
I20260619 08:23:16.524942 26334 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad3cad486ceb4d4abba0c261c09af0ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:23:16.524968 26334 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad3cad486ceb4d4abba0c261c09af0ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad3cad486ceb4d4abba0c261c09af0ca, State: Initialized, Role: FOLLOWER
I20260619 08:23:16.525118 26334 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad3cad486ceb4d4abba0c261c09af0ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3cad486ceb4d4abba0c261c09af0ca" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44609 } }
I20260619 08:23:16.525216 26334 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad3cad486ceb4d4abba0c261c09af0ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:23:16.525242 26334 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad3cad486ceb4d4abba0c261c09af0ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:23:16.525274 26334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad3cad486ceb4d4abba0c261c09af0ca [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:23:16.526259 26334 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad3cad486ceb4d4abba0c261c09af0ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3cad486ceb4d4abba0c261c09af0ca" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44609 } }
I20260619 08:23:16.526355 26334 leader_election.cc:302] T 00000000000000000000000000000000 P ad3cad486ceb4d4abba0c261c09af0ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad3cad486ceb4d4abba0c261c09af0ca; no voters:
I20260619 08:23:16.526489 26334 leader_election.cc:288] T 00000000000000000000000000000000 P ad3cad486ceb4d4abba0c261c09af0ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:23:16.526566 26337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad3cad486ceb4d4abba0c261c09af0ca [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:23:16.526755 26337 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad3cad486ceb4d4abba0c261c09af0ca [term 1 LEADER]: Becoming Leader. State: Replica: ad3cad486ceb4d4abba0c261c09af0ca, State: Running, Role: LEADER
I20260619 08:23:16.526793 26334 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad3cad486ceb4d4abba0c261c09af0ca [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:23:16.526856 26337 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad3cad486ceb4d4abba0c261c09af0ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3cad486ceb4d4abba0c261c09af0ca" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44609 } }
I20260619 08:23:16.527307 26338 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad3cad486ceb4d4abba0c261c09af0ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad3cad486ceb4d4abba0c261c09af0ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3cad486ceb4d4abba0c261c09af0ca" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44609 } } }
I20260619 08:23:16.527454 26338 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad3cad486ceb4d4abba0c261c09af0ca [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:16.527588 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 26266
I20260619 08:23:16.527333 26339 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad3cad486ceb4d4abba0c261c09af0ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad3cad486ceb4d4abba0c261c09af0ca. Latest consensus state: current_term: 1 leader_uuid: "ad3cad486ceb4d4abba0c261c09af0ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3cad486ceb4d4abba0c261c09af0ca" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 44609 } } }
I20260619 08:23:16.527647 26339 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad3cad486ceb4d4abba0c261c09af0ca [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:16.527731 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:23:16.527796 26342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:23:16.527922 4702 external_mini_cluster.cc:1428] Setting key 39353a8caaa9272a90bcbf774a991393
I20260619 08:23:16.528249 26342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:23:16.530004 26342 catalog_manager.cc:1347] Generated new cluster ID: a4b87aa997bf4abb95b0f2aad4d0d555
I20260619 08:23:16.530061 26342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:23:16 dist-test-slave-d3n8 krb5kdc[26253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:23:16.536798 26342 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:23:16.537460 26342 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:23:16.541637 26350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:16.528646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38043 (local address 127.4.151.190:44609)
0619 08:23:16.529689 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:16.529699 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:16.529731 (+ 32us) server_negotiation.cc:407] Connection header received
0619 08:23:16.529855 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:16.529863 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:16.529925 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:16.530072 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:16.530847 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:16.532386 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:16.533542 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:16.534263 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:16.534383 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:16.537445 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:16.537449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:16.537450 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:16.537464 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:16.539657 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:16.540144 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:16.540148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:16.540148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:16.540270 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:16.540520 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:16.540521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:16.540522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:16.540806 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:16.540988 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:16.541124 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 08:23:16.541352 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":979,"thread_start_us":65,"threads_started":1}
I20260619 08:23:16.542683 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:23:16.549482 26342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad3cad486ceb4d4abba0c261c09af0ca: Generated new TSK 0
I20260619 08:23:16.549698 26342 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:23:16.552201 26350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:16.544172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56571 (local address 127.4.151.190:44609)
0619 08:23:16.544215 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:16.544217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:16.544391 (+ 174us) server_negotiation.cc:407] Connection header received
0619 08:23:16.544440 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:16.544441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:16.544451 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:16.544507 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:16.545127 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:16.546028 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:16.547080 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:16.547432 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:16.547499 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:16.549117 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:16.549119 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:16.549119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:16.549126 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:16.550738 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:16.551211 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:16.551211 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:16.551212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:16.551303 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:16.551534 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:16.551535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:16.551535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:16.551644 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:16.551733 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:16.551954 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 08:23:16.552074 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 08:23:16.554157 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 26266
W20260619 08:23:16.558172 26251 connection.cc:535] client connection to 127.4.151.190:44609 recv error: Network error: failed to read from TLS socket (remote: 127.4.151.190:44609): Connection reset by peer (error 104)
2026-06-19T08:23:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:23:16 dist-test-slave-d3n8 krb5kdc[26365](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:23:16 dist-test-slave-d3n8 krb5kdc[26365](info): set up 2 sockets
Jun 19 08:23:16 dist-test-slave-d3n8 krb5kdc[26365](info): commencing operation
krb5kdc: starting...
W20260619 08:23:17.597662 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:23:17 dist-test-slave-d3n8 krb5kdc[26365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:23:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:23:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:23:17.722622 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:38071
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:41871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:38071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:23:17.804936 26378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:23:17.805311 26378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:23:17.805356 26378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:23:17.806653 26378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:23:17.806708 26378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:23:17.806723 26378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:23:17.806737 26378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:23:17.806749 26378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:23:17.808429 26378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:41871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:38071
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:38071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:23:17.808741 26378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:23:17.808995 26378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:23:17.814229 26383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:17.814318 26386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:17.814368 26384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:17.814445 26378 server_base.cc:1034] running on GCE node
I20260619 08:23:17.814767 26378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:23:17.815081 26378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:23:17.816265 26378 hybrid_clock.cc:648] HybridClock initialized: now 1781857397816245 us; error 44 us; skew 500 ppm
Jun 19 08:23:17 dist-test-slave-d3n8 krb5kdc[26365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857397, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:23:17.824232 26378 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:23:17.824808 26378 webserver.cc:460] Webserver started at http://127.4.151.190:35185/ using document root <none> and password file <none>
I20260619 08:23:17.825050 26378 fs_manager.cc:359] Metadata directory not provided
I20260619 08:23:17.825120 26378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:23:17.825274 26378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:23:17.826398 26378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "516cb6b9029643148db3dfdde7baf0ad"
format_stamp: "Formatted at 2026-06-19 08:23:17 on dist-test-slave-d3n8"
server_key: "25e487166519690736463dbfab08c156"
server_key_iv: "744c3189b1f1f9b50fc1879ecfe0b151"
server_key_version: "encryptionkey@0"
I20260619 08:23:17.826771 26378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "516cb6b9029643148db3dfdde7baf0ad"
format_stamp: "Formatted at 2026-06-19 08:23:17 on dist-test-slave-d3n8"
server_key: "25e487166519690736463dbfab08c156"
server_key_iv: "744c3189b1f1f9b50fc1879ecfe0b151"
server_key_version: "encryptionkey@0"
I20260619 08:23:17.828145 26378 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 08:23:17.829092 26393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:23:17.829484 26378 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:23:17.829589 26378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "516cb6b9029643148db3dfdde7baf0ad"
format_stamp: "Formatted at 2026-06-19 08:23:17 on dist-test-slave-d3n8"
server_key: "25e487166519690736463dbfab08c156"
server_key_iv: "744c3189b1f1f9b50fc1879ecfe0b151"
server_key_version: "encryptionkey@0"
I20260619 08:23:17.829679 26378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:23:17.846326 26378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:23:17.848691 26378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:23:17.848867 26378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:23:17.853220 26378 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:38071
I20260619 08:23:17.853260 26445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:38071 every 8 connection(s)
I20260619 08:23:17.853704 26378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:23:17.854619 26446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:23:17.856945 26446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 516cb6b9029643148db3dfdde7baf0ad: Bootstrap starting.
I20260619 08:23:17.857613 26446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 516cb6b9029643148db3dfdde7baf0ad: Neither blocks nor log segments found. Creating new log.
I20260619 08:23:17.857805 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 26378
I20260619 08:23:17.857930 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:23:17.857970 26446 log.cc:824] T 00000000000000000000000000000000 P 516cb6b9029643148db3dfdde7baf0ad: Log is configured to *not* fsync() on all Append() calls
I20260619 08:23:17.858145 4702 external_mini_cluster.cc:1428] Setting key 0fcead3c4f33432d1c6c17958122eb7c
I20260619 08:23:17.858915 26446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 516cb6b9029643148db3dfdde7baf0ad: No bootstrap required, opened a new log
I20260619 08:23:17.861291 26446 raft_consensus.cc:348] T 00000000000000000000000000000000 P 516cb6b9029643148db3dfdde7baf0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516cb6b9029643148db3dfdde7baf0ad" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38071 } }
I20260619 08:23:17.861555 26446 raft_consensus.cc:374] T 00000000000000000000000000000000 P 516cb6b9029643148db3dfdde7baf0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:23:17.861608 26446 raft_consensus.cc:729] T 00000000000000000000000000000000 P 516cb6b9029643148db3dfdde7baf0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb6b9029643148db3dfdde7baf0ad, State: Initialized, Role: FOLLOWER
I20260619 08:23:17.861747 26446 consensus_queue.cc:260] T 00000000000000000000000000000000 P 516cb6b9029643148db3dfdde7baf0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516cb6b9029643148db3dfdde7baf0ad" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38071 } }
I20260619 08:23:17.861852 26446 raft_consensus.cc:388] T 00000000000000000000000000000000 P 516cb6b9029643148db3dfdde7baf0ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:23:17.861897 26446 raft_consensus.cc:482] T 00000000000000000000000000000000 P 516cb6b9029643148db3dfdde7baf0ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:23:17.861948 26446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 516cb6b9029643148db3dfdde7baf0ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:23:17.862557 26446 raft_consensus.cc:504] T 00000000000000000000000000000000 P 516cb6b9029643148db3dfdde7baf0ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516cb6b9029643148db3dfdde7baf0ad" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38071 } }
I20260619 08:23:17.862696 26446 leader_election.cc:302] T 00000000000000000000000000000000 P 516cb6b9029643148db3dfdde7baf0ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 516cb6b9029643148db3dfdde7baf0ad; no voters:
I20260619 08:23:17.862872 26446 leader_election.cc:288] T 00000000000000000000000000000000 P 516cb6b9029643148db3dfdde7baf0ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:23:17.862922 26451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 516cb6b9029643148db3dfdde7baf0ad [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:23:17.863083 26451 raft_consensus.cc:686] T 00000000000000000000000000000000 P 516cb6b9029643148db3dfdde7baf0ad [term 1 LEADER]: Becoming Leader. State: Replica: 516cb6b9029643148db3dfdde7baf0ad, State: Running, Role: LEADER
I20260619 08:23:17.863145 26446 sys_catalog.cc:564] T 00000000000000000000000000000000 P 516cb6b9029643148db3dfdde7baf0ad [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:23:17.863200 26451 consensus_queue.cc:237] T 00000000000000000000000000000000 P 516cb6b9029643148db3dfdde7baf0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516cb6b9029643148db3dfdde7baf0ad" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38071 } }
I20260619 08:23:17.863575 26452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 516cb6b9029643148db3dfdde7baf0ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "516cb6b9029643148db3dfdde7baf0ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516cb6b9029643148db3dfdde7baf0ad" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38071 } } }
I20260619 08:23:17.863961 26452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 516cb6b9029643148db3dfdde7baf0ad [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:17.863973 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:17.858690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59783 (local address 127.4.151.190:38071)
0619 08:23:17.858893 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:17.858907 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:17.858933 (+ 26us) server_negotiation.cc:407] Connection header received
0619 08:23:17.859409 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:17.859416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:17.859462 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:17.859612 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:17.859847 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:17.860752 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:17.861638 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:17.862208 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:17.862348 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:17.863189 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:17.863193 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:17.863194 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:23:17.863447 (+ 253us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:17.863494 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:23:17.863511 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:17.863576 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:17.863733 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 08:23:17.863767 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":95,"threads_started":1}
I20260619 08:23:17.863561 26451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 516cb6b9029643148db3dfdde7baf0ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 516cb6b9029643148db3dfdde7baf0ad. Latest consensus state: current_term: 1 leader_uuid: "516cb6b9029643148db3dfdde7baf0ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516cb6b9029643148db3dfdde7baf0ad" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38071 } } }
I20260619 08:23:17.864225 26451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 516cb6b9029643148db3dfdde7baf0ad [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:17.865094 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 08:23:17.865751 26466 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 516cb6b9029643148db3dfdde7baf0ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:23:17.865811 26466 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:23:17.865857 26456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:23:17.866008 26456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:23:17.867498 26456 catalog_manager.cc:1347] Generated new cluster ID: 07b51f6c4b3345d3b25abdc41423f230
I20260619 08:23:17.867549 26456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:23:17.870361 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:17.866650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36083 (local address 127.4.151.190:38071)
0619 08:23:17.866687 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:17.866689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:17.866820 (+ 131us) server_negotiation.cc:407] Connection header received
0619 08:23:17.866893 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:17.866894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:17.866905 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:17.866952 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:17.867162 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:17.867891 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:17.868960 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:17.869284 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:17.869362 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:17.870105 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:17.870106 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:17.870106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:23:17.870112 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:17.870127 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:23:17.870132 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:17.870201 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:17.870272 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 08:23:17.870283 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:23:17.889760 26456 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:23:17.890350 26456 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:23:17.898499 26456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 516cb6b9029643148db3dfdde7baf0ad: Generated new TSK 0
I20260619 08:23:17.898749 26456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:23:17.911912 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 26378
W20260619 08:23:17.916253 26363 connection.cc:535] client connection to 127.4.151.190:38071 recv error: Network error: failed to read from TLS socket (remote: 127.4.151.190:38071): Connection reset by peer (error 104)
2026-06-19T08:23:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1357 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:23:17 dist-test-slave-d3n8 krb5kdc[26477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:23:17 dist-test-slave-d3n8 krb5kdc[26477](info): set up 2 sockets
Jun 19 08:23:17 dist-test-slave-d3n8 krb5kdc[26477](info): commencing operation
krb5kdc: starting...
W20260619 08:23:18.953686 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:23:19 dist-test-slave-d3n8 krb5kdc[26477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:23:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:23:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:23:19.081461 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:37787
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:40067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:37787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:23:19.163230 26490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:23:19.163501 26490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:23:19.163544 26490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:23:19.164729 26490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:23:19.164808 26490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:23:19.164836 26490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:23:19.164870 26490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:23:19.164897 26490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:23:19.166501 26490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:40067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:37787
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:37787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:23:19.166795 26490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:23:19.167032 26490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:23:19.172116 26498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:19.172231 26495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:19.172231 26496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:19.172271 26490 server_base.cc:1034] running on GCE node
I20260619 08:23:19.172817 26490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:23:19.173112 26490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:23:19.174309 26490 hybrid_clock.cc:648] HybridClock initialized: now 1781857399174296 us; error 53 us; skew 500 ppm
Jun 19 08:23:19 dist-test-slave-d3n8 krb5kdc[26477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857399, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:23:19.177078 26490 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:23:19.177608 26490 webserver.cc:460] Webserver started at http://127.4.151.190:43261/ using document root <none> and password file <none>
I20260619 08:23:19.177840 26490 fs_manager.cc:359] Metadata directory not provided
I20260619 08:23:19.177909 26490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:23:19.178062 26490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:23:19.179176 26490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "5b3e56f09ca8482c92800feb93bee49d"
format_stamp: "Formatted at 2026-06-19 08:23:19 on dist-test-slave-d3n8"
server_key: "5d18fbc7c29db613f4c98970f6ff42c4"
server_key_iv: "f819b14ba7d25e11b8b656216b1b5e3d"
server_key_version: "encryptionkey@0"
I20260619 08:23:19.179545 26490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "5b3e56f09ca8482c92800feb93bee49d"
format_stamp: "Formatted at 2026-06-19 08:23:19 on dist-test-slave-d3n8"
server_key: "5d18fbc7c29db613f4c98970f6ff42c4"
server_key_iv: "f819b14ba7d25e11b8b656216b1b5e3d"
server_key_version: "encryptionkey@0"
I20260619 08:23:19.180907 26490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:23:19.181710 26505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:23:19.181887 26490 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:23:19.181991 26490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "5b3e56f09ca8482c92800feb93bee49d"
format_stamp: "Formatted at 2026-06-19 08:23:19 on dist-test-slave-d3n8"
server_key: "5d18fbc7c29db613f4c98970f6ff42c4"
server_key_iv: "f819b14ba7d25e11b8b656216b1b5e3d"
server_key_version: "encryptionkey@0"
I20260619 08:23:19.182078 26490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:23:19.185953 26490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:23:19.188265 26490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:23:19.188437 26490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:23:19.192529 26490 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:37787
I20260619 08:23:19.192565 26557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:37787 every 8 connection(s)
I20260619 08:23:19.192952 26490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:23:19.194111 26558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:23:19.196453 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 26490
I20260619 08:23:19.196557 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:23:19.196781 4702 external_mini_cluster.cc:1428] Setting key 7732d1ede8b79c39dee3a35adcd568ee
I20260619 08:23:19.196864 26558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b3e56f09ca8482c92800feb93bee49d: Bootstrap starting.
I20260619 08:23:19.197691 26558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b3e56f09ca8482c92800feb93bee49d: Neither blocks nor log segments found. Creating new log.
I20260619 08:23:19.198014 26558 log.cc:824] T 00000000000000000000000000000000 P 5b3e56f09ca8482c92800feb93bee49d: Log is configured to *not* fsync() on all Append() calls
I20260619 08:23:19.198844 26558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b3e56f09ca8482c92800feb93bee49d: No bootstrap required, opened a new log
I20260619 08:23:19.199539 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:19.197319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60499 (local address 127.4.151.190:37787)
0619 08:23:19.197531 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:19.197538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:19.197563 (+ 25us) server_negotiation.cc:407] Connection header received
0619 08:23:19.198066 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:19.198073 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:19.198098 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:19.198365 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:19.199095 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:19.199100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:19.199100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:23:19.199110 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:19.199156 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:23:19.199171 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:19.199272 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:19.199311 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 08:23:19.199352 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":115,"threads_started":1}
I20260619 08:23:19.200498 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:23:19.201479 26558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b3e56f09ca8482c92800feb93bee49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3e56f09ca8482c92800feb93bee49d" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 37787 } }
I20260619 08:23:19.201670 26558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b3e56f09ca8482c92800feb93bee49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:23:19.201716 26558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b3e56f09ca8482c92800feb93bee49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b3e56f09ca8482c92800feb93bee49d, State: Initialized, Role: FOLLOWER
I20260619 08:23:19.201874 26558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b3e56f09ca8482c92800feb93bee49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3e56f09ca8482c92800feb93bee49d" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 37787 } }
I20260619 08:23:19.201995 26558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b3e56f09ca8482c92800feb93bee49d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:23:19.202042 26558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b3e56f09ca8482c92800feb93bee49d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:23:19.202106 26558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b3e56f09ca8482c92800feb93bee49d [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:23:19.202816 26558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b3e56f09ca8482c92800feb93bee49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3e56f09ca8482c92800feb93bee49d" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 37787 } }
I20260619 08:23:19.202939 26558 leader_election.cc:302] T 00000000000000000000000000000000 P 5b3e56f09ca8482c92800feb93bee49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b3e56f09ca8482c92800feb93bee49d; no voters:
I20260619 08:23:19.203117 26558 leader_election.cc:288] T 00000000000000000000000000000000 P 5b3e56f09ca8482c92800feb93bee49d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:23:19.203176 26568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b3e56f09ca8482c92800feb93bee49d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:23:19.203320 26568 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b3e56f09ca8482c92800feb93bee49d [term 1 LEADER]: Becoming Leader. State: Replica: 5b3e56f09ca8482c92800feb93bee49d, State: Running, Role: LEADER
I20260619 08:23:19.203404 26558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b3e56f09ca8482c92800feb93bee49d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:23:19.203477 26568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b3e56f09ca8482c92800feb93bee49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3e56f09ca8482c92800feb93bee49d" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 37787 } }
I20260619 08:23:19.203644 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:19.202299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48259 (local address 127.4.151.190:37787)
0619 08:23:19.202368 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:19.202370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:19.202499 (+ 129us) server_negotiation.cc:407] Connection header received
0619 08:23:19.202573 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:19.202574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:19.202579 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:19.202627 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:19.203336 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:19.203337 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:19.203337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:23:19.203343 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:19.203356 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:23:19.203361 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:19.203393 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:19.203533 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 08:23:19.203548 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 08:23:19.203889 26568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b3e56f09ca8482c92800feb93bee49d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b3e56f09ca8482c92800feb93bee49d. Latest consensus state: current_term: 1 leader_uuid: "5b3e56f09ca8482c92800feb93bee49d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3e56f09ca8482c92800feb93bee49d" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 37787 } } }
I20260619 08:23:19.203987 26568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b3e56f09ca8482c92800feb93bee49d [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:19.203904 26569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b3e56f09ca8482c92800feb93bee49d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b3e56f09ca8482c92800feb93bee49d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3e56f09ca8482c92800feb93bee49d" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 37787 } } }
I20260619 08:23:19.204197 26569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b3e56f09ca8482c92800feb93bee49d [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:19.204290 26575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:23:19.204775 26575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:23:19.206717 26575 catalog_manager.cc:1347] Generated new cluster ID: 8d8e4dab5f50417388c22c1318f17165
I20260619 08:23:19.206772 26575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:23:19.226028 26575 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:23:19.226228 26575 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:23:19.235548 26575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b3e56f09ca8482c92800feb93bee49d: Generated new TSK 0
I20260619 08:23:19.235767 26575 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:23:19.242775 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 26490
2026-06-19T08:23:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:23:19 dist-test-slave-d3n8 krb5kdc[26589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:23:19 dist-test-slave-d3n8 krb5kdc[26589](info): set up 2 sockets
Jun 19 08:23:19 dist-test-slave-d3n8 krb5kdc[26589](info): commencing operation
krb5kdc: starting...
W20260619 08:23:20.313613 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:23:20 dist-test-slave-d3n8 krb5kdc[26589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:23:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:23:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:23:20.436975 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:33449
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:40153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:33449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:23:20.517055 26602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:23:20.517323 26602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:23:20.517364 26602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:23:20.518554 26602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:23:20.518626 26602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:23:20.518651 26602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:23:20.518679 26602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:23:20.518710 26602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:23:20.520192 26602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:40153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:33449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:33449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:23:20.520488 26602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:23:20.520715 26602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:23:20.525588 26608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:20.525568 26610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:20.525568 26607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:20.525882 26602 server_base.cc:1034] running on GCE node
I20260619 08:23:20.526103 26602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:23:20.526396 26602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:23:20.527560 26602 hybrid_clock.cc:648] HybridClock initialized: now 1781857400527551 us; error 44 us; skew 500 ppm
Jun 19 08:23:20 dist-test-slave-d3n8 krb5kdc[26589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857400, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:23:20.530264 26602 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:23:20.530745 26602 webserver.cc:460] Webserver started at http://127.4.151.190:40659/ using document root <none> and password file <none>
I20260619 08:23:20.530989 26602 fs_manager.cc:359] Metadata directory not provided
I20260619 08:23:20.531056 26602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:23:20.531208 26602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:23:20.532272 26602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "b5899913720d48e98c685f1c22d7ebdd"
format_stamp: "Formatted at 2026-06-19 08:23:20 on dist-test-slave-d3n8"
server_key: "a45c3b3d9d3b2fba163a250c0a58a322"
server_key_iv: "dda2b8869ba73608b83d442a813525a4"
server_key_version: "encryptionkey@0"
I20260619 08:23:20.532650 26602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "b5899913720d48e98c685f1c22d7ebdd"
format_stamp: "Formatted at 2026-06-19 08:23:20 on dist-test-slave-d3n8"
server_key: "a45c3b3d9d3b2fba163a250c0a58a322"
server_key_iv: "dda2b8869ba73608b83d442a813525a4"
server_key_version: "encryptionkey@0"
I20260619 08:23:20.534128 26602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:23:20.534952 26617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:23:20.535312 26602 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:23:20.535423 26602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "b5899913720d48e98c685f1c22d7ebdd"
format_stamp: "Formatted at 2026-06-19 08:23:20 on dist-test-slave-d3n8"
server_key: "a45c3b3d9d3b2fba163a250c0a58a322"
server_key_iv: "dda2b8869ba73608b83d442a813525a4"
server_key_version: "encryptionkey@0"
I20260619 08:23:20.535503 26602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:23:20.550971 26602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:23:20.552947 26602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:23:20.553112 26602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:23:20.557144 26602 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:33449
I20260619 08:23:20.557190 26669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:33449 every 8 connection(s)
I20260619 08:23:20.557585 26602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:23:20.558509 26670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:23:20.560797 26670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5899913720d48e98c685f1c22d7ebdd: Bootstrap starting.
I20260619 08:23:20.561445 26670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5899913720d48e98c685f1c22d7ebdd: Neither blocks nor log segments found. Creating new log.
I20260619 08:23:20.561755 26670 log.cc:824] T 00000000000000000000000000000000 P b5899913720d48e98c685f1c22d7ebdd: Log is configured to *not* fsync() on all Append() calls
I20260619 08:23:20.562045 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 26602
I20260619 08:23:20.562177 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:23:20.562404 4702 external_mini_cluster.cc:1428] Setting key 8e761117b71105903c100f2620728908
I20260619 08:23:20.562638 26670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5899913720d48e98c685f1c22d7ebdd: No bootstrap required, opened a new log
I20260619 08:23:20.564395 26670 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5899913720d48e98c685f1c22d7ebdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5899913720d48e98c685f1c22d7ebdd" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33449 } }
I20260619 08:23:20.564559 26670 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5899913720d48e98c685f1c22d7ebdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:23:20.564648 26670 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5899913720d48e98c685f1c22d7ebdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5899913720d48e98c685f1c22d7ebdd, State: Initialized, Role: FOLLOWER
I20260619 08:23:20.564801 26670 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5899913720d48e98c685f1c22d7ebdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5899913720d48e98c685f1c22d7ebdd" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33449 } }
I20260619 08:23:20.564894 26670 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5899913720d48e98c685f1c22d7ebdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:23:20.564931 26670 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5899913720d48e98c685f1c22d7ebdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:23:20.564983 26670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5899913720d48e98c685f1c22d7ebdd [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:23:20.566006 26670 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5899913720d48e98c685f1c22d7ebdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5899913720d48e98c685f1c22d7ebdd" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33449 } }
I20260619 08:23:20.566098 26670 leader_election.cc:302] T 00000000000000000000000000000000 P b5899913720d48e98c685f1c22d7ebdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5899913720d48e98c685f1c22d7ebdd; no voters:
I20260619 08:23:20.566298 26670 leader_election.cc:288] T 00000000000000000000000000000000 P b5899913720d48e98c685f1c22d7ebdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:23:20.566363 26675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5899913720d48e98c685f1c22d7ebdd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:23:20.566496 26675 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5899913720d48e98c685f1c22d7ebdd [term 1 LEADER]: Becoming Leader. State: Replica: b5899913720d48e98c685f1c22d7ebdd, State: Running, Role: LEADER
I20260619 08:23:20.566563 26670 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5899913720d48e98c685f1c22d7ebdd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:23:20.566640 26675 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5899913720d48e98c685f1c22d7ebdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5899913720d48e98c685f1c22d7ebdd" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33449 } }
I20260619 08:23:20.567073 26676 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5899913720d48e98c685f1c22d7ebdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5899913720d48e98c685f1c22d7ebdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5899913720d48e98c685f1c22d7ebdd" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33449 } } }
I20260619 08:23:20.567147 26676 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5899913720d48e98c685f1c22d7ebdd [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:20.567343 26677 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5899913720d48e98c685f1c22d7ebdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5899913720d48e98c685f1c22d7ebdd. Latest consensus state: current_term: 1 leader_uuid: "b5899913720d48e98c685f1c22d7ebdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5899913720d48e98c685f1c22d7ebdd" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 33449 } } }
I20260619 08:23:20.567440 26677 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5899913720d48e98c685f1c22d7ebdd [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:20.567708 26683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:23:20.568192 26683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:23:20.569834 26683 catalog_manager.cc:1347] Generated new cluster ID: 4a0bd58d99a44b41a0bbf43d045cab42
I20260619 08:23:20.569882 26683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:23:20 dist-test-slave-d3n8 krb5kdc[26589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:23:20.574860 26674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:20.562956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38413 (local address 127.4.151.190:33449)
0619 08:23:20.563126 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:20.563134 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:20.563169 (+ 35us) server_negotiation.cc:407] Connection header received
0619 08:23:20.564387 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:20.564395 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:20.564448 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:20.564611 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:20.565417 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:20.566532 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:20.567664 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:20.568265 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:20.568779 (+ 514us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:20.571289 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:20.571292 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:20.571293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:20.571303 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:20.573179 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:20.573615 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:20.573618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:20.573619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:20.573742 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:20.573956 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:20.573957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:20.573957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:20.574245 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:20.574386 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:20.574532 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 08:23:20.574664 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260619 08:23:20.575654 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:23:20.580910 26683 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:23:20.581634 26683 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:23:20.584395 26674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:20.577045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44653 (local address 127.4.151.190:33449)
0619 08:23:20.577076 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:20.577078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:20.577202 (+ 124us) server_negotiation.cc:407] Connection header received
0619 08:23:20.577230 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:20.577231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:20.577242 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:20.577311 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:20.577929 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:20.578680 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:20.579655 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:20.579996 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:20.580069 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:20.581482 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:20.581484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:20.581484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:20.581491 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:20.583217 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:20.583602 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:20.583603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:20.583603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:20.583663 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:20.583871 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:20.583872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:20.583872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:20.584007 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:20.584107 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:20.584205 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 08:23:20.584303 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 08:23:20.586699 26683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5899913720d48e98c685f1c22d7ebdd: Generated new TSK 0
I20260619 08:23:20.586889 26683 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:23:20.592370 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 26602
2026-06-19T08:23:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1349 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:23:20 dist-test-slave-d3n8 krb5kdc[26701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:23:20 dist-test-slave-d3n8 krb5kdc[26701](info): set up 2 sockets
Jun 19 08:23:20 dist-test-slave-d3n8 krb5kdc[26701](info): commencing operation
krb5kdc: starting...
W20260619 08:23:21.669742 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:23:21 dist-test-slave-d3n8 krb5kdc[26701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:23:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:23:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:23:21.801578 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:38191
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:44941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:38191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:23:21.884492 26714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:23:21.884763 26714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:23:21.884809 26714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:23:21.886248 26714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:23:21.886327 26714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:23:21.886356 26714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:23:21.886389 26714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:23:21.886417 26714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:23:21.888103 26714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:44941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:38191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:38191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:23:21.888406 26714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:23:21.888666 26714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:23:21.894204 26720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:21.894465 26714 server_base.cc:1034] running on GCE node
W20260619 08:23:21.894191 26719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:21.894187 26722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:21.894846 26714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:23:21.895164 26714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:23:21.896353 26714 hybrid_clock.cc:648] HybridClock initialized: now 1781857401896327 us; error 61 us; skew 500 ppm
Jun 19 08:23:21 dist-test-slave-d3n8 krb5kdc[26701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857401, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:23:21.898917 26714 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:23:21.899341 26714 webserver.cc:460] Webserver started at http://127.4.151.190:39335/ using document root <none> and password file <none>
I20260619 08:23:21.899569 26714 fs_manager.cc:359] Metadata directory not provided
I20260619 08:23:21.899610 26714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:23:21.899766 26714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:23:21.900827 26714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "78bebed267054fa3995e0e89cc777508"
format_stamp: "Formatted at 2026-06-19 08:23:21 on dist-test-slave-d3n8"
server_key: "604357a8c72ab86aa50c79500e1e2581"
server_key_iv: "3abab3a9a66cd9e03b43e36e25460a18"
server_key_version: "encryptionkey@0"
I20260619 08:23:21.901185 26714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "78bebed267054fa3995e0e89cc777508"
format_stamp: "Formatted at 2026-06-19 08:23:21 on dist-test-slave-d3n8"
server_key: "604357a8c72ab86aa50c79500e1e2581"
server_key_iv: "3abab3a9a66cd9e03b43e36e25460a18"
server_key_version: "encryptionkey@0"
I20260619 08:23:21.902558 26714 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:23:21.903446 26729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:23:21.903702 26714 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:23:21.903815 26714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "78bebed267054fa3995e0e89cc777508"
format_stamp: "Formatted at 2026-06-19 08:23:21 on dist-test-slave-d3n8"
server_key: "604357a8c72ab86aa50c79500e1e2581"
server_key_iv: "3abab3a9a66cd9e03b43e36e25460a18"
server_key_version: "encryptionkey@0"
I20260619 08:23:21.903908 26714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:23:21.925609 26714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:23:21.928128 26714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:23:21.928306 26714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:23:21.932596 26714 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:38191
I20260619 08:23:21.932657 26781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:38191 every 8 connection(s)
I20260619 08:23:21.933034 26714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:23:21.933972 26782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:23:21.936347 26782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78bebed267054fa3995e0e89cc777508: Bootstrap starting.
I20260619 08:23:21.936805 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 26714
I20260619 08:23:21.936899 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:23:21.937043 26782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78bebed267054fa3995e0e89cc777508: Neither blocks nor log segments found. Creating new log.
I20260619 08:23:21.937151 4702 external_mini_cluster.cc:1428] Setting key 4a697d82ed0092408f26537a24340fab
I20260619 08:23:21.937397 26782 log.cc:824] T 00000000000000000000000000000000 P 78bebed267054fa3995e0e89cc777508: Log is configured to *not* fsync() on all Append() calls
I20260619 08:23:21.938228 26782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78bebed267054fa3995e0e89cc777508: No bootstrap required, opened a new log
I20260619 08:23:21.939848 26782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78bebed267054fa3995e0e89cc777508 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bebed267054fa3995e0e89cc777508" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38191 } }
I20260619 08:23:21.940012 26782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78bebed267054fa3995e0e89cc777508 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:23:21.940060 26782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78bebed267054fa3995e0e89cc777508 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78bebed267054fa3995e0e89cc777508, State: Initialized, Role: FOLLOWER
I20260619 08:23:21.940227 26782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78bebed267054fa3995e0e89cc777508 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bebed267054fa3995e0e89cc777508" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38191 } }
I20260619 08:23:21.940322 26782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78bebed267054fa3995e0e89cc777508 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:23:21.940368 26782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78bebed267054fa3995e0e89cc777508 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:23:21.940418 26782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78bebed267054fa3995e0e89cc777508 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:23:21.941148 26782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78bebed267054fa3995e0e89cc777508 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bebed267054fa3995e0e89cc777508" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38191 } }
I20260619 08:23:21.941257 26782 leader_election.cc:302] T 00000000000000000000000000000000 P 78bebed267054fa3995e0e89cc777508 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78bebed267054fa3995e0e89cc777508; no voters:
I20260619 08:23:21.941435 26782 leader_election.cc:288] T 00000000000000000000000000000000 P 78bebed267054fa3995e0e89cc777508 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:23:21.941478 26787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78bebed267054fa3995e0e89cc777508 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:23:21.941637 26787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78bebed267054fa3995e0e89cc777508 [term 1 LEADER]: Becoming Leader. State: Replica: 78bebed267054fa3995e0e89cc777508, State: Running, Role: LEADER
I20260619 08:23:21.941717 26782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78bebed267054fa3995e0e89cc777508 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:23:21.941782 26787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78bebed267054fa3995e0e89cc777508 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bebed267054fa3995e0e89cc777508" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38191 } }
I20260619 08:23:21.942112 26787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78bebed267054fa3995e0e89cc777508 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78bebed267054fa3995e0e89cc777508. Latest consensus state: current_term: 1 leader_uuid: "78bebed267054fa3995e0e89cc777508" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bebed267054fa3995e0e89cc777508" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38191 } } }
I20260619 08:23:21.942121 26788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78bebed267054fa3995e0e89cc777508 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78bebed267054fa3995e0e89cc777508" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bebed267054fa3995e0e89cc777508" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38191 } } }
I20260619 08:23:21.942260 26787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78bebed267054fa3995e0e89cc777508 [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:21.942281 26788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78bebed267054fa3995e0e89cc777508 [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:21.942587 26795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:23:21.943117 26795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:23:21.943832 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:21.937775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43075 (local address 127.4.151.190:38191)
0619 08:23:21.937956 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:21.937965 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:21.938001 (+ 36us) server_negotiation.cc:407] Connection header received
0619 08:23:21.938533 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:21.938540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:21.938588 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:21.938700 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:21.938955 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:21.940390 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:21.941565 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:21.942232 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:21.942690 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:21.943083 (+ 393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:21.943087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:21.943087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:23:21.943294 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:21.943333 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:23:21.943349 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:21.943419 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:21.943534 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 08:23:21.943606 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260619 08:23:21.944973 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:23:21.949529 26795 catalog_manager.cc:1347] Generated new cluster ID: 9f2b7018ac6c4f4fb2edd4a2e14c0fe7
I20260619 08:23:21.949612 26795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:23:21.950469 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:21.946551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40221 (local address 127.4.151.190:38191)
0619 08:23:21.946612 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:21.946614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:21.946642 (+ 28us) server_negotiation.cc:407] Connection header received
0619 08:23:21.946717 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:21.946718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:21.946731 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:21.946792 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:21.946947 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:21.947685 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:21.948491 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:21.948814 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:21.948895 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:21.950071 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:21.950072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:21.950073 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:23:21.950078 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:21.950095 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:23:21.950100 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:21.950152 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:21.950358 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 08:23:21.950383 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 08:23:21.958782 26795 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:23:21.959354 26795 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:23:21.965570 26795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78bebed267054fa3995e0e89cc777508: Generated new TSK 0
I20260619 08:23:21.965772 26795 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:23:21.971917 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 26714
2026-06-19T08:23:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:23:22 dist-test-slave-d3n8 krb5kdc[26813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:23:22 dist-test-slave-d3n8 krb5kdc[26813](info): set up 2 sockets
Jun 19 08:23:22 dist-test-slave-d3n8 krb5kdc[26813](info): commencing operation
krb5kdc: starting...
W20260619 08:23:23.109609 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:23:23 dist-test-slave-d3n8 krb5kdc[26813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:23:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:23:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:23:23.234476 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:37837
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:40525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:37837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:23:23.321878 26826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:23:23.322094 26826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:23:23.322119 26826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:23:23.323315 26826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:23:23.323364 26826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:23:23.323377 26826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:23:23.323390 26826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:23:23.323403 26826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:23:23.324920 26826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:40525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:37837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:37837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:23:23.325193 26826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:23:23.325484 26826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:23:23.330899 26832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:23.330904 26831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:23.330937 26826 server_base.cc:1034] running on GCE node
W20260619 08:23:23.330912 26834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:23.331409 26826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:23:23.331713 26826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:23:23.332908 26826 hybrid_clock.cc:648] HybridClock initialized: now 1781857403332899 us; error 34 us; skew 500 ppm
Jun 19 08:23:23 dist-test-slave-d3n8 krb5kdc[26813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857403, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:23:23.341636 26826 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:23:23.342167 26826 webserver.cc:460] Webserver started at http://127.4.151.190:41783/ using document root <none> and password file <none>
I20260619 08:23:23.342446 26826 fs_manager.cc:359] Metadata directory not provided
I20260619 08:23:23.342517 26826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:23:23.342679 26826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:23:23.343811 26826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "2b20a9a5f6d342ec9e656a3e91127e8c"
format_stamp: "Formatted at 2026-06-19 08:23:23 on dist-test-slave-d3n8"
server_key: "28bafef559c1f598f43a3ab59d12be23"
server_key_iv: "5aaaea428663793c5e91c338011aa55f"
server_key_version: "encryptionkey@0"
I20260619 08:23:23.344200 26826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "2b20a9a5f6d342ec9e656a3e91127e8c"
format_stamp: "Formatted at 2026-06-19 08:23:23 on dist-test-slave-d3n8"
server_key: "28bafef559c1f598f43a3ab59d12be23"
server_key_iv: "5aaaea428663793c5e91c338011aa55f"
server_key_version: "encryptionkey@0"
I20260619 08:23:23.345747 26826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:23:23.346617 26841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:23:23.346894 26826 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:23:23.347007 26826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "2b20a9a5f6d342ec9e656a3e91127e8c"
format_stamp: "Formatted at 2026-06-19 08:23:23 on dist-test-slave-d3n8"
server_key: "28bafef559c1f598f43a3ab59d12be23"
server_key_iv: "5aaaea428663793c5e91c338011aa55f"
server_key_version: "encryptionkey@0"
I20260619 08:23:23.347087 26826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:23:23.350684 26826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:23:23.354051 26826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:23:23.354387 26826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:23:23.359153 26826 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:37837
I20260619 08:23:23.359196 26893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:37837 every 8 connection(s)
I20260619 08:23:23.359643 26826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:23:23.360744 26894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:23:23.363170 26894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b20a9a5f6d342ec9e656a3e91127e8c: Bootstrap starting.
I20260619 08:23:23.363782 26894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b20a9a5f6d342ec9e656a3e91127e8c: Neither blocks nor log segments found. Creating new log.
I20260619 08:23:23.364071 26894 log.cc:824] T 00000000000000000000000000000000 P 2b20a9a5f6d342ec9e656a3e91127e8c: Log is configured to *not* fsync() on all Append() calls
I20260619 08:23:23.364818 26894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b20a9a5f6d342ec9e656a3e91127e8c: No bootstrap required, opened a new log
I20260619 08:23:23.366569 26894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b20a9a5f6d342ec9e656a3e91127e8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b20a9a5f6d342ec9e656a3e91127e8c" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 37837 } }
I20260619 08:23:23.366721 26894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b20a9a5f6d342ec9e656a3e91127e8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:23:23.366744 26894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b20a9a5f6d342ec9e656a3e91127e8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b20a9a5f6d342ec9e656a3e91127e8c, State: Initialized, Role: FOLLOWER
I20260619 08:23:23.366904 26894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b20a9a5f6d342ec9e656a3e91127e8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b20a9a5f6d342ec9e656a3e91127e8c" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 37837 } }
I20260619 08:23:23.366969 26894 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b20a9a5f6d342ec9e656a3e91127e8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:23:23.366991 26894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b20a9a5f6d342ec9e656a3e91127e8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:23:23.367064 26894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b20a9a5f6d342ec9e656a3e91127e8c [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:23:23.368047 26894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b20a9a5f6d342ec9e656a3e91127e8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b20a9a5f6d342ec9e656a3e91127e8c" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 37837 } }
I20260619 08:23:23.368144 26894 leader_election.cc:302] T 00000000000000000000000000000000 P 2b20a9a5f6d342ec9e656a3e91127e8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b20a9a5f6d342ec9e656a3e91127e8c; no voters:
I20260619 08:23:23.368327 26894 leader_election.cc:288] T 00000000000000000000000000000000 P 2b20a9a5f6d342ec9e656a3e91127e8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:23:23.368398 26897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b20a9a5f6d342ec9e656a3e91127e8c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:23:23.368602 26894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b20a9a5f6d342ec9e656a3e91127e8c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:23:23.368701 26897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b20a9a5f6d342ec9e656a3e91127e8c [term 1 LEADER]: Becoming Leader. State: Replica: 2b20a9a5f6d342ec9e656a3e91127e8c, State: Running, Role: LEADER
I20260619 08:23:23.368860 26897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b20a9a5f6d342ec9e656a3e91127e8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b20a9a5f6d342ec9e656a3e91127e8c" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 37837 } }
I20260619 08:23:23.369268 26898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b20a9a5f6d342ec9e656a3e91127e8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b20a9a5f6d342ec9e656a3e91127e8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b20a9a5f6d342ec9e656a3e91127e8c" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 37837 } } }
I20260619 08:23:23.369354 26898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b20a9a5f6d342ec9e656a3e91127e8c [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:23.369611 26899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b20a9a5f6d342ec9e656a3e91127e8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b20a9a5f6d342ec9e656a3e91127e8c. Latest consensus state: current_term: 1 leader_uuid: "2b20a9a5f6d342ec9e656a3e91127e8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b20a9a5f6d342ec9e656a3e91127e8c" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 37837 } } }
I20260619 08:23:23.369728 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 26826
I20260619 08:23:23.369745 26899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b20a9a5f6d342ec9e656a3e91127e8c [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:23.369863 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:23:23.370090 4702 external_mini_cluster.cc:1428] Setting key 0290d4df73ebdfb2de10109fb7389409
W20260619 08:23:23.370520 26913 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2b20a9a5f6d342ec9e656a3e91127e8c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:23:23.370600 26913 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:23:23.370664 26906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:23:23.370813 26906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:23:23.372362 26916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:23.370553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35477 (local address 127.4.151.190:37837)
0619 08:23:23.370884 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:23.370893 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:23.370928 (+ 35us) server_negotiation.cc:407] Connection header received
0619 08:23:23.371038 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:23.371043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:23.371069 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:23.371205 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:23.371900 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:23.371904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:23.371904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:23:23.371909 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:23.371972 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:23:23.371984 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:23.372062 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:23.372134 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 08:23:23.372177 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":86,"threads_started":1}
I20260619 08:23:23.372637 26906 catalog_manager.cc:1347] Generated new cluster ID: 38431fdd726e44e291eb3103bd779c30
I20260619 08:23:23.372678 26906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:23:23.373355 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:23:23.379007 26916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:23.377699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44605 (local address 127.4.151.190:37837)
0619 08:23:23.377744 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:23.377746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:23.377929 (+ 183us) server_negotiation.cc:407] Connection header received
0619 08:23:23.377966 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:23.377966 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:23.377973 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:23.378043 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:23.378770 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:23.378771 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:23.378772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:23:23.378777 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:23.378795 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:23:23.378799 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:23.378862 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:23.378916 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 08:23:23.378934 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:23:23.380874 26906 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:23:23.381036 26906 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:23:23.385699 26906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b20a9a5f6d342ec9e656a3e91127e8c: Generated new TSK 0
I20260619 08:23:23.385926 26906 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:23:23.389921 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 26826
2026-06-19T08:23:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1418 ms)
[----------] 18 tests from AuthTokenIssuingTest (24394 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:23:23 dist-test-slave-d3n8 krb5kdc[26925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:23:23 dist-test-slave-d3n8 krb5kdc[26925](info): set up 2 sockets
Jun 19 08:23:23 dist-test-slave-d3n8 krb5kdc[26925](info): commencing operation
krb5kdc: starting...
W20260619 08:23:24.441699 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:23:24 dist-test-slave-d3n8 krb5kdc[26925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:23:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:23:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:23:24.569033 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:36787
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:36787,127.4.151.189:39933,127.4.151.188:33847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:23:24.654302 26938 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 08:23:24.654536 26938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:23:24.654552 26938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:23:24.654573 26938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:23:24.655784 26938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:23:24.655833 26938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:23:24.655848 26938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:23:24.655859 26938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:23:24.655872 26938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:23:24.657920 26938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:36787,127.4.151.189:39933,127.4.151.188:33847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:36787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:23:24.658265 26938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:23:24.658560 26938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:23:24.663662 26938 server_base.cc:1034] running on GCE node
W20260619 08:23:24.663573 26946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:24.663630 26944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:24.663573 26943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:24.664091 26938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:23:24.664470 26938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:23:24.665668 26938 hybrid_clock.cc:648] HybridClock initialized: now 1781857404665657 us; error 38 us; skew 500 ppm
Jun 19 08:23:24 dist-test-slave-d3n8 krb5kdc[26925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857404, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:23:24.668100 26938 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:23:24.668565 26938 webserver.cc:460] Webserver started at http://127.4.151.190:35247/ using document root <none> and password file <none>
I20260619 08:23:24.668776 26938 fs_manager.cc:359] Metadata directory not provided
I20260619 08:23:24.668818 26938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:23:24.668907 26938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:23:24.669999 26938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "9d3b04cd2b5d4727b6726b37feaa236e"
format_stamp: "Formatted at 2026-06-19 08:23:24 on dist-test-slave-d3n8"
server_key: "4c9d963c565ed21b33a59b98acbc8a9a"
server_key_iv: "c5d408a467fb7eef01bd47cca3122030"
server_key_version: "encryptionkey@0"
I20260619 08:23:24.670305 26938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "9d3b04cd2b5d4727b6726b37feaa236e"
format_stamp: "Formatted at 2026-06-19 08:23:24 on dist-test-slave-d3n8"
server_key: "4c9d963c565ed21b33a59b98acbc8a9a"
server_key_iv: "c5d408a467fb7eef01bd47cca3122030"
server_key_version: "encryptionkey@0"
I20260619 08:23:24.671551 26938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:23:24.672369 26953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:23:24.672561 26938 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:23:24.672657 26938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "9d3b04cd2b5d4727b6726b37feaa236e"
format_stamp: "Formatted at 2026-06-19 08:23:24 on dist-test-slave-d3n8"
server_key: "4c9d963c565ed21b33a59b98acbc8a9a"
server_key_iv: "c5d408a467fb7eef01bd47cca3122030"
server_key_version: "encryptionkey@0"
I20260619 08:23:24.672740 26938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:23:24.687214 26938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:23:24.689633 26938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:23:24.689798 26938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:23:24.693821 26938 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:36787
I20260619 08:23:24.693848 27005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:36787 every 8 connection(s)
I20260619 08:23:24.694254 26938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:23:24.695477 27006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:23:24.697108 27006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.190" port: 36787 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 08:23:24 dist-test-slave-d3n8 krb5kdc[26925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857404, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:23:24.704110 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 26938
I20260619 08:23:24.704210 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:23:24.704407 4702 external_mini_cluster.cc:1428] Setting key 66b7bc167c74f831198fb1b28696a0b0
I20260619 08:23:24.707803 27009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:24.697904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57346 (local address 127.4.151.190:36787)
0619 08:23:24.698080 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:24.698090 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:24.698235 (+ 145us) server_negotiation.cc:407] Connection header received
0619 08:23:24.698738 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:24.698740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:24.698757 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:24.698843 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:24.699379 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:24.700337 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:24.701281 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:24.701701 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:24.701814 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:24.703946 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:24.703952 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:24.703953 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:24.703959 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:24.706131 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:24.706549 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:24.706552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:24.706553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:24.706638 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:24.706882 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:24.706883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:24.706883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:24.707111 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:24.707202 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:24.707405 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 08:23:24.707548 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":112,"threads_started":1}
I20260619 08:23:24.707805 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:24.697867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:36787 (local address 127.0.0.1:57346)
0619 08:23:24.698048 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:24.698105 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 08:23:24.698565 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:23:24.698847 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:23:24.698849 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:23:24.699101 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:23:24.699259 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:24.699263 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:24.700435 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:24.700437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:24.701159 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:24.701161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:24.701820 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:24.701821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:24.701880 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:24.702361 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:23:24.702369 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:23:24.703846 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:23:24.706234 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:24.706239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:24.706242 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:24.706441 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:24.706711 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:24.706711 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:24.706712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:24.706807 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:24.707193 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:23:24.707198 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:23:24.707261 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 08:23:24.707371 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 08:23:24.707621 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":94,"thread_start_us":98,"threads_started":1}
I20260619 08:23:24.709004 27006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.189" port: 39933 } has no permanent_uuid. Determining permanent_uuid...
I20260619 08:23:24.709478 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:24.709278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:39933 (local address 127.0.0.1:35308)
0619 08:23:24.709348 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:24.709439 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.189:39933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 08:23:24.709649 26956 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.151.189:39933: connect: Connection refused (error 111)
I20260619 08:23:24.710139 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:24.710028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:39933 (local address 127.0.0.1:35314)
0619 08:23:24.710079 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:24.710113 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.189:39933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 08:23:24.710368 27006 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.151.189:39933: Network error: Client connection negotiation failed: client connection to 127.4.151.189:39933: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.4.151.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.189.keytab.
Entry for principal kudu/127.4.151.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.189.keytab.
I20260619 08:23:24.733662 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.189:39933
--webserver_interface=127.4.151.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.189.keytab
--principal=kudu/127.4.151.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:36787,127.4.151.189:39933,127.4.151.188:33847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:23:24.759627 27006 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.151.189" port: 39933 } attempt: 1
I20260619 08:23:24.760452 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:24.760185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:39933 (local address 127.0.0.1:35320)
0619 08:23:24.760358 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:24.760412 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.189:39933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260619 08:23:24.760962 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:24.760836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:39933 (local address 127.0.0.1:35334)
0619 08:23:24.760890 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:24.760929 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.189:39933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 08:23:24.761116 27006 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.151.189:39933: Network error: Client connection negotiation failed: client connection to 127.4.151.189:39933: connect: Connection refused (error 111)
W20260619 08:23:24.816718 27013 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 08:23:24.816953 27013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:23:24.816972 27013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:23:24.816995 27013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:23:24.818327 27013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:23:24.818383 27013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:23:24.818398 27013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:23:24.818411 27013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:23:24.818424 27013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:23:24.820317 27013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:36787,127.4.151.189:39933,127.4.151.188:33847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.189.keytab
--principal=kudu/127.4.151.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.189:39933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.151.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:23:24.820638 27013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:23:24.820905 27013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:23:24.826251 27018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:24.826220 27021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:24.826414 27013 server_base.cc:1034] running on GCE node
W20260619 08:23:24.826305 27019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:24.826748 27013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:23:24.827134 27013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:23:24.828311 27013 hybrid_clock.cc:648] HybridClock initialized: now 1781857404828277 us; error 53 us; skew 500 ppm
I20260619 08:23:24.829303 27006 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.151.189" port: 39933 } attempt: 2
I20260619 08:23:24.829766 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:24.829597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:39933 (local address 127.0.0.1:35340)
0619 08:23:24.829690 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:24.829732 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.189:39933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 08:23:24.830235 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:24.830101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:39933 (local address 127.0.0.1:35350)
0619 08:23:24.830178 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:24.830209 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.189:39933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 08:23:24.830370 27006 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.151.189:39933: Network error: Client connection negotiation failed: client connection to 127.4.151.189:39933: connect: Connection refused (error 111)
Jun 19 08:23:24 dist-test-slave-d3n8 krb5kdc[26925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857404, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:23:24.831420 27013 init.cc:377] Logged in from keytab as kudu/127.4.151.189@KRBTEST.COM (short username kudu)
I20260619 08:23:24.831869 27013 webserver.cc:460] Webserver started at http://127.4.151.189:38967/ using document root <none> and password file <none>
I20260619 08:23:24.832134 27013 fs_manager.cc:359] Metadata directory not provided
I20260619 08:23:24.832206 27013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:23:24.832370 27013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:23:24.833544 27013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-1/data/instance:
uuid: "d5bb5aff69cb4e1cb1c5d0d3b55bf27a"
format_stamp: "Formatted at 2026-06-19 08:23:24 on dist-test-slave-d3n8"
server_key: "b2561fbe8245a07f9979a3b01147572e"
server_key_iv: "387db0e510e9a6576e64ca7457ea009d"
server_key_version: "encryptionkey@0"
I20260619 08:23:24.833963 27013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-1/wal/instance:
uuid: "d5bb5aff69cb4e1cb1c5d0d3b55bf27a"
format_stamp: "Formatted at 2026-06-19 08:23:24 on dist-test-slave-d3n8"
server_key: "b2561fbe8245a07f9979a3b01147572e"
server_key_iv: "387db0e510e9a6576e64ca7457ea009d"
server_key_version: "encryptionkey@0"
I20260619 08:23:24.835455 27013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:23:24.836284 27028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:23:24.836634 27013 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:23:24.836764 27013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-1/wal
uuid: "d5bb5aff69cb4e1cb1c5d0d3b55bf27a"
format_stamp: "Formatted at 2026-06-19 08:23:24 on dist-test-slave-d3n8"
server_key: "b2561fbe8245a07f9979a3b01147572e"
server_key_iv: "387db0e510e9a6576e64ca7457ea009d"
server_key_version: "encryptionkey@0"
I20260619 08:23:24.836863 27013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:23:24.852494 27013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:23:24.855007 27013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:23:24.855216 27013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:23:24.859607 27013 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.189:39933
I20260619 08:23:24.859725 27080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.189:39933 every 8 connection(s)
I20260619 08:23:24.860056 27013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-1/data/info.pb
I20260619 08:23:24.861088 27081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:23:24.862797 27081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.190" port: 36787 } has no permanent_uuid. Determining permanent_uuid...
I20260619 08:23:24.869028 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 27013
Jun 19 08:23:24 dist-test-slave-d3n8 krb5kdc[26925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857404, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.189@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:23:24.869145 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-1/wal/instance
I20260619 08:23:24.869410 4702 external_mini_cluster.cc:1428] Setting key 987c3594a86f8a55b353899a3b6d7d04
I20260619 08:23:24.872691 27009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:24.863629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57356 (local address 127.4.151.190:36787)
0619 08:23:24.863783 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:24.863785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:24.863961 (+ 176us) server_negotiation.cc:407] Connection header received
0619 08:23:24.864595 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:24.864596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:24.864613 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:24.864679 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:24.865320 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:24.865952 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:24.867006 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:24.867224 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:24.867311 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:24.869740 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:24.869745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:24.869746 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:24.869752 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:24.871223 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:24.871678 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:24.871680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:24.871680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:24.871743 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:24.872005 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:24.872005 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:24.872006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:24.872131 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:24.872225 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:24.872496 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 08:23:24.872588 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260619 08:23:24.872952 27083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:24.863543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:36787 (local address 127.0.0.1:57356)
0619 08:23:24.863836 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:24.863873 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 08:23:24.864471 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:23:24.864758 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:23:24.864762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:23:24.865004 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:23:24.865212 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:24.865215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:24.866069 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:24.866076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:24.866897 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:24.866899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:24.867336 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:24.867338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:24.867472 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:24.868018 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:23:24.868026 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:23:24.869612 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:23:24.871332 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:24.871340 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:24.871342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:24.871578 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:24.871817 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:24.871817 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:24.871818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:24.871935 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:24.872199 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:23:24.872206 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:23:24.872357 (+ 151us) client_negotiation.cc:769] Sending connection context
0619 08:23:24.872477 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 08:23:24.872645 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":139,"threads_started":1}
I20260619 08:23:24.873574 27081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.189" port: 39933 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.151.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.188@KRBTEST.COM" created.
Jun 19 08:23:24 dist-test-slave-d3n8 krb5kdc[26925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857404, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.189@KRBTEST.COM for kudu/127.4.151.189@KRBTEST.COM
I20260619 08:23:24.885529 27083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:24.873897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:39933 (local address 127.0.0.1:35362)
0619 08:23:24.873944 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:24.873957 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:23:24.874041 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:23:24.874338 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:23:24.874340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:23:24.874519 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:23:24.874598 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:24.874600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:24.875586 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:24.875588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:24.876440 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:24.876442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:24.877056 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:24.877057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:24.877147 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:24.877609 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:23:24.877616 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:23:24.881715 (+ 4099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:23:24.884124 (+ 2409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:24.884125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:24.884126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:24.884434 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:24.884686 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:24.884687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:24.884688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:24.884761 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:24.885094 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:23:24.885095 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:23:24.885150 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 08:23:24.885234 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 08:23:24.885361 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 08:23:24.885672 27085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:24.873940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35362 (local address 127.4.151.189:39933)
0619 08:23:24.874145 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:24.874150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:24.874174 (+ 24us) server_negotiation.cc:407] Connection header received
0619 08:23:24.874199 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:24.874201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:24.874219 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:24.874312 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:24.874701 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:24.875502 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:24.876618 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:24.876964 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:24.877028 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:24.881848 (+ 4820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:24.881860 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:24.881860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:24.881869 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:24.884022 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:24.884544 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:24.884547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:24.884547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:24.884610 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:24.884848 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:24.884849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:24.884849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:24.885016 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:24.885086 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:24.885363 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 08:23:24.885506 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":149,"threads_started":1}
I20260619 08:23:24.886399 27081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } has no permanent_uuid. Determining permanent_uuid...
I20260619 08:23:24.886814 27083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:24.886682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:33847 (local address 127.0.0.1:53934)
0619 08:23:24.886740 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:24.886786 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 08:23:24.886945 27032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.151.188:33847: connect: Connection refused (error 111)
I20260619 08:23:24.887437 27083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:24.887340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:33847 (local address 127.0.0.1:53936)
0619 08:23:24.887391 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:24.887414 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 08:23:24.887549 27081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.151.188:33847: Network error: Client connection negotiation failed: client connection to 127.4.151.188:33847: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.188.keytab.
Entry for principal kudu/127.4.151.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.188.keytab.
I20260619 08:23:24.902112 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.188:33847
--webserver_interface=127.4.151.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.188.keytab
--principal=kudu/127.4.151.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:35717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:36787,127.4.151.189:39933,127.4.151.188:33847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:23:24.921542 27006 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.151.189" port: 39933 } attempt: 3
Jun 19 08:23:24 dist-test-slave-d3n8 krb5kdc[26925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857404, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for kudu/127.4.151.189@KRBTEST.COM
I20260619 08:23:24.929127 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:24.921995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:39933 (local address 127.0.0.1:35368)
0619 08:23:24.922053 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:24.922092 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 08:23:24.922148 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:23:24.922441 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:23:24.922442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:23:24.922610 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:23:24.922715 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:24.922717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:24.923397 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:24.923398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:24.923934 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:24.923936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:24.924344 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:24.924345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:24.924426 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:24.924942 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:23:24.924946 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:23:24.926269 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:23:24.928082 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:24.928082 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:24.928083 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:24.928307 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:24.928533 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:24.928533 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:24.928534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:24.928591 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:24.928870 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:23:24.928871 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:23:24.928915 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 08:23:24.928972 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 08:23:24.929054 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 08:23:24.929183 27085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:24.922060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.4.151.189:39933)
0619 08:23:24.922112 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:24.922114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:24.922129 (+ 15us) server_negotiation.cc:407] Connection header received
0619 08:23:24.922346 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:24.922348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:24.922360 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:24.922435 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:24.922803 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:24.923313 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:24.924023 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:24.924267 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:24.924346 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:24.926366 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:24.926368 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:24.926368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:24.926377 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:24.927987 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:24.928385 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:24.928386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:24.928387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:24.928448 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:24.928657 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:24.928657 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:24.928658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:24.928802 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:24.928870 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:24.929015 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 08:23:24.929093 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 08:23:24.929639 27006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } has no permanent_uuid. Determining permanent_uuid...
I20260619 08:23:24.930042 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:24.929933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:33847 (local address 127.0.0.1:53938)
0619 08:23:24.929984 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:24.930012 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 08:23:24.930406 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:24.930327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:33847 (local address 127.0.0.1:53952)
0619 08:23:24.930369 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:24.930391 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 08:23:24.930511 27006 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.151.188:33847: Network error: Client connection negotiation failed: client connection to 127.4.151.188:33847: connect: Connection refused (error 111)
I20260619 08:23:24.936791 27081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } attempt: 1
I20260619 08:23:24.937237 27083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:24.937103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:33847 (local address 127.0.0.1:53968)
0619 08:23:24.937166 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:24.937206 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 08:23:24.937616 27083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:24.937514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:33847 (local address 127.0.0.1:53976)
0619 08:23:24.937563 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:24.937595 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 08:23:24.937717 27081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.151.188:33847: Network error: Client connection negotiation failed: client connection to 127.4.151.188:33847: connect: Connection refused (error 111)
I20260619 08:23:24.961676 27006 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } attempt: 1
I20260619 08:23:24.962327 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:24.962108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:33847 (local address 127.0.0.1:53984)
0619 08:23:24.962234 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:24.962281 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260619 08:23:24.962770 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:24.962674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:33847 (local address 127.0.0.1:53996)
0619 08:23:24.962719 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:24.962747 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 08:23:24.962894 27006 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.151.188:33847: Network error: Client connection negotiation failed: client connection to 127.4.151.188:33847: connect: Connection refused (error 111)
W20260619 08:23:24.987564 27088 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 08:23:24.987783 27088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:23:24.987802 27088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:23:24.987823 27088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:23:24.988925 27088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:23:24.988962 27088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:23:24.988977 27088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:23:24.988989 27088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:23:24.989001 27088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:23:24.990545 27088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:35717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:36787,127.4.151.189:39933,127.4.151.188:33847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.188.keytab
--principal=kudu/127.4.151.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.188:33847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.151.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:23:24.990754 27088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:23:24.990922 27088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:23:24.996157 27094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:24.996188 27096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:24.996425 27088 server_base.cc:1034] running on GCE node
W20260619 08:23:24.996469 27093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:24.996831 27088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:23:24.997143 27088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:23:24.998335 27088 hybrid_clock.cc:648] HybridClock initialized: now 1781857404998334 us; error 44 us; skew 500 ppm
Jun 19 08:23:24 dist-test-slave-d3n8 krb5kdc[26925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857404, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:23:25.000988 27088 init.cc:377] Logged in from keytab as kudu/127.4.151.188@KRBTEST.COM (short username kudu)
I20260619 08:23:25.001567 27088 webserver.cc:460] Webserver started at http://127.4.151.188:36281/ using document root <none> and password file <none>
I20260619 08:23:25.001770 27088 fs_manager.cc:359] Metadata directory not provided
I20260619 08:23:25.001812 27088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:23:25.001904 27088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:23:25.002990 27088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-2/data/instance:
uuid: "f031e664ecd7401fa8575f53e2f6a2cd"
format_stamp: "Formatted at 2026-06-19 08:23:24 on dist-test-slave-d3n8"
server_key: "5966bbe1bd5433f97a002ad3c27c8cd8"
server_key_iv: "01f07854e5e94d34edc7f9323b16f47a"
server_key_version: "encryptionkey@0"
I20260619 08:23:25.003309 27088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-2/wal/instance:
uuid: "f031e664ecd7401fa8575f53e2f6a2cd"
format_stamp: "Formatted at 2026-06-19 08:23:24 on dist-test-slave-d3n8"
server_key: "5966bbe1bd5433f97a002ad3c27c8cd8"
server_key_iv: "01f07854e5e94d34edc7f9323b16f47a"
server_key_version: "encryptionkey@0"
I20260619 08:23:25.004783 27088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:23:25.005787 27103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:23:25.005914 27081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } attempt: 2
I20260619 08:23:25.005995 27088 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:23:25.006183 27088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-2/wal
uuid: "f031e664ecd7401fa8575f53e2f6a2cd"
format_stamp: "Formatted at 2026-06-19 08:23:24 on dist-test-slave-d3n8"
server_key: "5966bbe1bd5433f97a002ad3c27c8cd8"
server_key_iv: "01f07854e5e94d34edc7f9323b16f47a"
server_key_version: "encryptionkey@0"
I20260619 08:23:25.006291 27088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:23:25.006438 27083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:25.006212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:33847 (local address 127.0.0.1:53998)
0619 08:23:25.006276 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:25.006370 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 08:23:25.006780 27083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:25.006668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:33847 (local address 127.0.0.1:54012)
0619 08:23:25.006724 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:25.006751 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 08:23:25.006886 27081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.151.188:33847: Network error: Client connection negotiation failed: client connection to 127.4.151.188:33847: connect: Connection refused (error 111)
I20260619 08:23:25.019302 27088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:23:25.021775 27088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:23:25.021890 27088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:23:25.025882 27088 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.188:33847
I20260619 08:23:25.025914 27155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.188:33847 every 8 connection(s)
I20260619 08:23:25.026280 27088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-2/data/info.pb
I20260619 08:23:25.027281 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 27088
I20260619 08:23:25.027365 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781857130600961-4702-0/minicluster-data/master-2/wal/instance
I20260619 08:23:25.027518 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:23:25.027590 4702 external_mini_cluster.cc:1428] Setting key 734c91cb977e19d3502a00f9e856a6f2
I20260619 08:23:25.029824 27156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.190" port: 36787 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 08:23:25 dist-test-slave-d3n8 krb5kdc[26925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.188@KRBTEST.COM
Jun 19 08:23:25 dist-test-slave-d3n8 krb5kdc[26925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857404, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.188@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:23:25.038028 27006 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } attempt: 2
I20260619 08:23:25.038499 27159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:25.028100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54018 (local address 127.4.151.188:33847)
0619 08:23:25.028306 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:25.028314 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:25.028339 (+ 25us) server_negotiation.cc:407] Connection header received
0619 08:23:25.028749 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:25.028756 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:25.028807 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:25.028955 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:25.029645 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.030793 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.031586 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.032002 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.032133 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:25.034894 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:25.034901 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:25.034901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:25.034907 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:25.036661 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:25.037075 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:25.037079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:25.037079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:25.037164 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:25.037475 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:25.037476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:25.037476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:25.037716 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:25.037875 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:25.037982 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 08:23:25.038316 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":91,"threads_started":1}
I20260619 08:23:25.038506 27009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:25.030619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57360 (local address 127.4.151.190:36787)
0619 08:23:25.030674 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:25.030676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:25.030836 (+ 160us) server_negotiation.cc:407] Connection header received
0619 08:23:25.030935 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:25.030936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:25.030948 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:25.031022 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:25.031531 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.031990 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.032793 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.032989 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.033076 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:25.035209 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:25.035210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:25.035210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:25.035218 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:25.037110 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:25.037656 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:25.037657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:25.037658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:25.037723 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:25.037923 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:25.037924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:25.037925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:25.038042 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:25.038116 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:25.038310 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 08:23:25.038404 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 08:23:25.039161 27160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:25.030521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:36787 (local address 127.0.0.1:57360)
0619 08:23:25.030711 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:25.030765 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 08:23:25.030867 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:23:25.031044 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:23:25.031047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:23:25.031341 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:23:25.031409 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:25.031411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.032085 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.032087 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:25.032699 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:25.032701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.033107 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.033109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:25.033182 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:25.033728 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:23:25.033740 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:23:25.035096 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:23:25.037324 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:25.037330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:25.037333 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:25.037575 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:25.037790 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:25.037791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:25.037791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:25.037858 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:25.038109 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:23:25.038115 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:23:25.038166 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 08:23:25.038807 (+ 641us) client_negotiation.cc:240] Negotiation successful
0619 08:23:25.038916 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":75,"threads_started":1}
I20260619 08:23:25.039731 27156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.189" port: 39933 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 08:23:25 dist-test-slave-d3n8 krb5kdc[26925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857404, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for kudu/127.4.151.188@KRBTEST.COM
Jun 19 08:23:25 dist-test-slave-d3n8 krb5kdc[26925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857404, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.188@KRBTEST.COM for kudu/127.4.151.189@KRBTEST.COM
Jun 19 08:23:25 dist-test-slave-d3n8 krb5kdc[26925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:23:25.047487 27160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:25.040020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:39933 (local address 127.0.0.1:35370)
0619 08:23:25.040058 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:25.040066 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 08:23:25.040097 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:23:25.040231 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:23:25.040232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:23:25.040402 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:23:25.040451 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:25.040453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.041335 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.041336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:25.042032 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:25.042035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.042448 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.042449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:25.042521 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:25.042853 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:23:25.042857 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:23:25.043981 (+ 1124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:23:25.046335 (+ 2354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:25.046336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:25.046337 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:25.046557 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:25.046797 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:25.046797 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:25.046797 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:25.046861 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:25.047136 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:23:25.047137 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:23:25.047194 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 08:23:25.047274 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 08:23:25.047384 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 08:23:25.047487 27085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:25.040078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.4.151.189:39933)
0619 08:23:25.040126 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:25.040128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:25.040137 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:23:25.040162 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:25.040163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:25.040171 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:25.040227 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:25.040538 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.041253 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.042121 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.042359 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.042447 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:25.044071 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:25.044072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:25.044073 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:25.044081 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:25.046138 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:25.046638 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:25.046639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:25.046640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:25.046707 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:25.046937 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:25.046938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:25.046938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:25.047061 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:25.047133 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:25.047313 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 08:23:25.047392 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 08:23:25.047641 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:25.038292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:33847 (local address 127.0.0.1:54022)
0619 08:23:25.038343 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:25.038353 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:23:25.038388 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:23:25.038748 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:23:25.038749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:23:25.038901 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:23:25.038944 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:25.038946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.039881 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.039882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:25.040661 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:25.040664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.041187 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.041188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:25.041249 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:25.041713 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:23:25.041718 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:23:25.043427 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:23:25.045240 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:25.045241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:25.045242 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:25.045974 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:25.046490 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:25.046491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:25.046491 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:25.046561 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:25.047069 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:23:25.047070 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:23:25.047119 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 08:23:25.047167 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 08:23:25.047480 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 08:23:25.048161 27156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } has no permanent_uuid. Determining permanent_uuid...
I20260619 08:23:25.048265 27159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:25.038460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54022 (local address 127.4.151.188:33847)
0619 08:23:25.038614 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:25.038616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:25.038625 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:23:25.038647 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:25.038649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:25.038659 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:25.038729 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:25.039022 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.039784 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.040782 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.041099 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.041420 (+ 321us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:25.043532 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:25.043533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:25.043533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:25.043540 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:25.045093 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:25.046064 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:25.046065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:25.046066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:25.046124 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:25.046798 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:25.046800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:25.046800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:25.046978 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:25.047873 (+ 895us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:25.047991 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 08:23:25.048147 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
I20260619 08:23:25.048835 27009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:25.039451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57370 (local address 127.4.151.190:36787)
0619 08:23:25.039484 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:25.039486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:25.039554 (+ 68us) server_negotiation.cc:407] Connection header received
0619 08:23:25.039607 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:25.039608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:25.039617 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:25.039671 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:25.040392 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.040893 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.041808 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.042076 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.042144 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:25.045458 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:25.045460 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:25.045460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:25.045466 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:25.047126 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:25.047597 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:25.047598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:25.047598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:25.047653 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:25.047873 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:25.047874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:25.047874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:25.048016 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:25.048293 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:25.048539 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 08:23:25.048672 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:23:25.050140 27006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e: Bootstrap starting.
I20260619 08:23:25.050900 27006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e: Neither blocks nor log segments found. Creating new log.
I20260619 08:23:25.051368 27006 log.cc:824] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e: Log is configured to *not* fsync() on all Append() calls
I20260619 08:23:25.052367 27006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e: No bootstrap required, opened a new log
Jun 19 08:23:25 dist-test-slave-d3n8 krb5kdc[26925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857404, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.188@KRBTEST.COM for kudu/127.4.151.188@KRBTEST.COM
Jun 19 08:23:25 dist-test-slave-d3n8 krb5kdc[26925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.189@KRBTEST.COM
I20260619 08:23:25.054958 27006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 36787 } } peers { permanent_uuid: "d5bb5aff69cb4e1cb1c5d0d3b55bf27a" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 39933 } } peers { permanent_uuid: "f031e664ecd7401fa8575f53e2f6a2cd" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } }
I20260619 08:23:25.055135 27006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d3b04cd2b5d4727b6726b37feaa236e, State: Initialized, Role: FOLLOWER
I20260619 08:23:25.055250 27006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 36787 } } peers { permanent_uuid: "d5bb5aff69cb4e1cb1c5d0d3b55bf27a" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 39933 } } peers { permanent_uuid: "f031e664ecd7401fa8575f53e2f6a2cd" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } }
I20260619 08:23:25.055702 27006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:23:25.056797 27160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:25.048548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:33847 (local address 127.0.0.1:54028)
0619 08:23:25.048645 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:25.048655 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:23:25.048692 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:23:25.049075 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:23:25.049076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:23:25.049217 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:23:25.049250 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:25.049251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.050149 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.050150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:25.050900 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:25.050902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.051508 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.051509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:25.051565 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:25.052023 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:23:25.052028 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:23:25.053276 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:23:25.055280 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:25.055281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:25.055282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:25.055482 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:25.055726 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:25.055726 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:25.055727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:25.055785 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:25.056391 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:23:25.056392 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:23:25.056440 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 08:23:25.056520 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 08:23:25.056620 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 08:23:25.057152 27159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:25.048617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54028 (local address 127.4.151.188:33847)
0619 08:23:25.048940 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:25.048942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:25.048954 (+ 12us) server_negotiation.cc:407] Connection header received
0619 08:23:25.048972 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:25.048973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:25.048983 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:25.049037 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:25.049391 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.050061 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.050990 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.051415 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.051503 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:25.053434 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:25.053435 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:25.053436 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:25.053444 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:25.055186 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:25.055556 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:25.055557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:25.055558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:25.055607 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:25.055847 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:25.055849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:25.055849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:25.056319 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:25.056404 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:25.056984 (+ 580us) server_negotiation.cc:299] Negotiation successful
0619 08:23:25.057055 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296}
I20260619 08:23:25.055653 27163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 36787 } } peers { permanent_uuid: "d5bb5aff69cb4e1cb1c5d0d3b55bf27a" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 39933 } } peers { permanent_uuid: "f031e664ecd7401fa8575f53e2f6a2cd" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } } }
I20260619 08:23:25.057814 27163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:23:25.058269 27085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:25.049518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35380 (local address 127.4.151.189:39933)
0619 08:23:25.049554 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:25.049556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:25.049564 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:23:25.049622 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:25.049623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:25.049632 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:25.049686 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:25.050175 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.050622 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.051288 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.051549 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.051614 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:25.054964 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:25.054966 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:25.054966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:25.054975 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:25.056828 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:25.057259 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:25.057260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:25.057261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:25.057321 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:25.057561 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:25.057561 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:25.057562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:25.057700 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:25.057854 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:25.058060 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 08:23:25.058176 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 08:23:25.058794 27156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f031e664ecd7401fa8575f53e2f6a2cd: Bootstrap starting.
I20260619 08:23:25.059911 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:23:25.059963 27156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f031e664ecd7401fa8575f53e2f6a2cd: Neither blocks nor log segments found. Creating new log.
I20260619 08:23:25.060277 27156 log.cc:824] T 00000000000000000000000000000000 P f031e664ecd7401fa8575f53e2f6a2cd: Log is configured to *not* fsync() on all Append() calls
I20260619 08:23:25.061044 26979 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9d3b04cd2b5d4727b6726b37feaa236e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57370
I20260619 08:23:25.061177 27156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f031e664ecd7401fa8575f53e2f6a2cd: No bootstrap required, opened a new log
I20260619 08:23:25.061194 26979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 08:23:25.061250 26979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e [term 0 FOLLOWER]: Advancing to term 1
W20260619 08:23:25.061616 27174 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:23:25.061733 27174 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:23:25.058879 26938 master_runner.cc:186] Error getting master registration for 127.4.151.189:39933: OK, instance_id { permanent_uuid: "d5bb5aff69cb4e1cb1c5d0d3b55bf27a" instance_seqno: 1781857404858297 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 08:23:25.062132 26938 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 08:23:25.062286 26979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 36787 } } peers { permanent_uuid: "d5bb5aff69cb4e1cb1c5d0d3b55bf27a" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 39933 } } peers { permanent_uuid: "f031e664ecd7401fa8575f53e2f6a2cd" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } }
I20260619 08:23:25.062541 26979 leader_election.cc:288] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e [CANDIDATE]: Term 1 election: Requested vote from peers d5bb5aff69cb4e1cb1c5d0d3b55bf27a (127.4.151.189:39933), f031e664ecd7401fa8575f53e2f6a2cd (127.4.151.188:33847)
I20260619 08:23:25.063524 27156 raft_consensus.cc:348] T 00000000000000000000000000000000 P f031e664ecd7401fa8575f53e2f6a2cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 36787 } } peers { permanent_uuid: "d5bb5aff69cb4e1cb1c5d0d3b55bf27a" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 39933 } } peers { permanent_uuid: "f031e664ecd7401fa8575f53e2f6a2cd" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } }
I20260619 08:23:25.063700 27156 raft_consensus.cc:729] T 00000000000000000000000000000000 P f031e664ecd7401fa8575f53e2f6a2cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f031e664ecd7401fa8575f53e2f6a2cd, State: Initialized, Role: FOLLOWER
I20260619 08:23:25.063792 27156 consensus_queue.cc:260] T 00000000000000000000000000000000 P f031e664ecd7401fa8575f53e2f6a2cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 36787 } } peers { permanent_uuid: "d5bb5aff69cb4e1cb1c5d0d3b55bf27a" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 39933 } } peers { permanent_uuid: "f031e664ecd7401fa8575f53e2f6a2cd" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } }
I20260619 08:23:25.064114 27156 sys_catalog.cc:564] T 00000000000000000000000000000000 P f031e664ecd7401fa8575f53e2f6a2cd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:23:25.064172 27130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f031e664ecd7401fa8575f53e2f6a2cd"
I20260619 08:23:25.064301 27130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f031e664ecd7401fa8575f53e2f6a2cd [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:23:25.065670 27130 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f031e664ecd7401fa8575f53e2f6a2cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d3b04cd2b5d4727b6726b37feaa236e in term 1.
I20260619 08:23:25.065948 26957 leader_election.cc:302] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d3b04cd2b5d4727b6726b37feaa236e, f031e664ecd7401fa8575f53e2f6a2cd; no voters:
I20260619 08:23:25.066087 27163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:23:25.066258 27163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e [term 1 LEADER]: Becoming Leader. State: Replica: 9d3b04cd2b5d4727b6726b37feaa236e, State: Running, Role: LEADER
I20260619 08:23:25.066366 27163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 36787 } } peers { permanent_uuid: "d5bb5aff69cb4e1cb1c5d0d3b55bf27a" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 39933 } } peers { permanent_uuid: "f031e664ecd7401fa8575f53e2f6a2cd" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } }
I20260619 08:23:25.066815 27163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d3b04cd2b5d4727b6726b37feaa236e. Latest consensus state: current_term: 1 leader_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 36787 } } peers { permanent_uuid: "d5bb5aff69cb4e1cb1c5d0d3b55bf27a" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 39933 } } peers { permanent_uuid: "f031e664ecd7401fa8575f53e2f6a2cd" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } } }
I20260619 08:23:25.066907 27163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:25.070125 27181 sys_catalog.cc:455] T 00000000000000000000000000000000 P f031e664ecd7401fa8575f53e2f6a2cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 36787 } } peers { permanent_uuid: "d5bb5aff69cb4e1cb1c5d0d3b55bf27a" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 39933 } } peers { permanent_uuid: "f031e664ecd7401fa8575f53e2f6a2cd" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } } }
I20260619 08:23:25.070451 27181 sys_catalog.cc:458] T 00000000000000000000000000000000 P f031e664ecd7401fa8575f53e2f6a2cd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:23:25.069324 27088 master_runner.cc:186] Error getting master registration for 127.4.151.189:39933: OK, instance_id { permanent_uuid: "d5bb5aff69cb4e1cb1c5d0d3b55bf27a" instance_seqno: 1781857404858297 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 08:23:25.072913 27088 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 08:23:25.074607 27009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:25.064002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57382 (local address 127.4.151.190:36787)
0619 08:23:25.064043 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:25.064045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:25.065273 (+ 1228us) server_negotiation.cc:407] Connection header received
0619 08:23:25.065466 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:25.065468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:25.065482 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:25.065518 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:25.066906 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.067706 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.069201 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.069537 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.069578 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:25.071354 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:25.071356 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:25.071356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:25.071364 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:25.073107 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:25.073610 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:25.073612 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:25.073612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:25.073674 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:25.073897 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:25.073898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:25.073898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:25.074042 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:25.074172 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:25.074413 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 08:23:25.074506 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 08:23:25.076280 27159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:25.064449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54032 (local address 127.4.151.188:33847)
0619 08:23:25.064496 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:25.064498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:25.065993 (+ 1495us) server_negotiation.cc:407] Connection header received
0619 08:23:25.066099 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:25.066099 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:25.066110 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:25.066142 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:25.067824 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.068547 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.069612 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.069933 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.069975 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:25.072406 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:25.072409 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:25.072410 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:25.072417 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:25.074453 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:25.075315 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:25.075316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:25.075316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:25.075398 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:25.075629 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:25.075630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:25.075631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:25.075777 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:25.075882 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:25.076054 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 08:23:25.076164 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 08:23:25.076788 27197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:23:25.077060 27197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:23:25.077472 27085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:25.064908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35396 (local address 127.4.151.189:39933)
0619 08:23:25.064956 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:25.064958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:25.066762 (+ 1804us) server_negotiation.cc:407] Connection header received
0619 08:23:25.067076 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:25.067078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:25.067090 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:25.067123 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:25.067806 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.068575 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.069689 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.070553 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.070603 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:25.073827 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:25.073829 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:25.073829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:25.073837 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:25.075916 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:25.076444 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:25.076445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:25.076445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:25.076504 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:25.076752 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:25.076753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:25.076754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:25.076956 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:25.077031 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:25.077237 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 08:23:25.077314 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 08:23:25.079911 27130 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f031e664ecd7401fa8575f53e2f6a2cd [term 1 FOLLOWER]: Refusing update from remote peer 9d3b04cd2b5d4727b6726b37feaa236e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:23:25.080250 27195 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f031e664ecd7401fa8575f53e2f6a2cd" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:23:25.081630 27181 sys_catalog.cc:455] T 00000000000000000000000000000000 P f031e664ecd7401fa8575f53e2f6a2cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d3b04cd2b5d4727b6726b37feaa236e. Latest consensus state: current_term: 1 leader_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 36787 } } peers { permanent_uuid: "d5bb5aff69cb4e1cb1c5d0d3b55bf27a" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 39933 } } peers { permanent_uuid: "f031e664ecd7401fa8575f53e2f6a2cd" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } } }
I20260619 08:23:25.081730 27181 sys_catalog.cc:458] T 00000000000000000000000000000000 P f031e664ecd7401fa8575f53e2f6a2cd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:23:25.082736 27163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 36787 } } peers { permanent_uuid: "d5bb5aff69cb4e1cb1c5d0d3b55bf27a" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 39933 } } peers { permanent_uuid: "f031e664ecd7401fa8575f53e2f6a2cd" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } } }
I20260619 08:23:25.082827 27163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:25.082906 27199 mvcc.cc:205] Tried to move back new op lower bound from 7298487931202818048 to 7298487931152621568. Current Snapshot: MvccSnapshot[applied={T|T < 7298487931202818048}]
I20260619 08:23:25.083213 27197 catalog_manager.cc:1347] Generated new cluster ID: 5e846cfef20e4d6f9bf4c7599dc730b0
I20260619 08:23:25.083320 27197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:23:25.083362 27181 sys_catalog.cc:455] T 00000000000000000000000000000000 P f031e664ecd7401fa8575f53e2f6a2cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 36787 } } peers { permanent_uuid: "d5bb5aff69cb4e1cb1c5d0d3b55bf27a" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 39933 } } peers { permanent_uuid: "f031e664ecd7401fa8575f53e2f6a2cd" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } } }
I20260619 08:23:25.083436 27181 sys_catalog.cc:458] T 00000000000000000000000000000000 P f031e664ecd7401fa8575f53e2f6a2cd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:23:25.089897 27198 catalog_manager.cc:1259] Loaded cluster ID: 5e846cfef20e4d6f9bf4c7599dc730b0
I20260619 08:23:25.089949 27198 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f031e664ecd7401fa8575f53e2f6a2cd: loading cluster ID for follower catalog manager: success
I20260619 08:23:25.090412 27197 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:23:25.091758 27197 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:23:25.092168 27198 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f031e664ecd7401fa8575f53e2f6a2cd: acquiring CA information for follower catalog manager: success
W20260619 08:23:25.092304 27198 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260619 08:23:25.098002 27081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } attempt: 3
I20260619 08:23:25.099009 27197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e: Generated new TSK 0
I20260619 08:23:25.099360 27197 catalog_manager.cc:1514] Initializing in-progress tserver states...
Jun 19 08:23:25 dist-test-slave-d3n8 krb5kdc[26925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857404, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.189@KRBTEST.COM for kudu/127.4.151.188@KRBTEST.COM
I20260619 08:23:25.110889 27083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:25.098320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:33847 (local address 127.0.0.1:54038)
0619 08:23:25.098452 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:25.098466 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 08:23:25.098507 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:23:25.098840 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:23:25.098842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:23:25.099043 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:23:25.099094 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:25.099097 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.100093 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.100094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:25.100960 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:25.100963 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.101768 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.101769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:25.101842 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:25.102605 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:23:25.102611 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:23:25.107126 (+ 4515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:23:25.109159 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:25.109160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:25.109161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:25.109363 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:25.109623 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:25.109624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:25.109624 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:25.109688 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:25.110435 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:23:25.110436 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:23:25.110516 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 08:23:25.110595 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 08:23:25.110750 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260619 08:23:25.111263 27159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:25.098578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54038 (local address 127.4.151.188:33847)
0619 08:23:25.098700 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:25.098702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:25.098710 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:23:25.098729 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:25.098729 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:25.098740 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:25.098789 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:25.099260 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.099985 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.101055 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:25.101334 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:25.101418 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:25.107234 (+ 5816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:25.107235 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:25.107235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:25.107244 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:25.109055 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:25.109476 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:25.109478 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:25.109479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:25.109541 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:25.109740 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:25.109741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:25.109742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:25.110052 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:25.110394 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:25.111076 (+ 682us) server_negotiation.cc:299] Negotiation successful
0619 08:23:25.111181 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260619 08:23:25.112824 27081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5bb5aff69cb4e1cb1c5d0d3b55bf27a: Bootstrap starting.
I20260619 08:23:25.113569 27081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5bb5aff69cb4e1cb1c5d0d3b55bf27a: Neither blocks nor log segments found. Creating new log.
I20260619 08:23:25.113874 27081 log.cc:824] T 00000000000000000000000000000000 P d5bb5aff69cb4e1cb1c5d0d3b55bf27a: Log is configured to *not* fsync() on all Append() calls
I20260619 08:23:25.114833 27081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5bb5aff69cb4e1cb1c5d0d3b55bf27a: No bootstrap required, opened a new log
I20260619 08:23:25.116976 27081 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5bb5aff69cb4e1cb1c5d0d3b55bf27a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 36787 } } peers { permanent_uuid: "d5bb5aff69cb4e1cb1c5d0d3b55bf27a" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 39933 } } peers { permanent_uuid: "f031e664ecd7401fa8575f53e2f6a2cd" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } }
I20260619 08:23:25.117125 27081 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5bb5aff69cb4e1cb1c5d0d3b55bf27a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5bb5aff69cb4e1cb1c5d0d3b55bf27a, State: Initialized, Role: FOLLOWER
I20260619 08:23:25.117218 27081 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5bb5aff69cb4e1cb1c5d0d3b55bf27a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 36787 } } peers { permanent_uuid: "d5bb5aff69cb4e1cb1c5d0d3b55bf27a" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 39933 } } peers { permanent_uuid: "f031e664ecd7401fa8575f53e2f6a2cd" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } }
I20260619 08:23:25.117549 27081 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5bb5aff69cb4e1cb1c5d0d3b55bf27a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:23:25.117671 27054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5bb5aff69cb4e1cb1c5d0d3b55bf27a [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:23:25.118757 27054 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d5bb5aff69cb4e1cb1c5d0d3b55bf27a [term 1 FOLLOWER]: Refusing update from remote peer 9d3b04cd2b5d4727b6726b37feaa236e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:23:25.119132 27163 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5bb5aff69cb4e1cb1c5d0d3b55bf27a" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:23:25.119508 27055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d5bb5aff69cb4e1cb1c5d0d3b55bf27a"
I20260619 08:23:25.120400 27055 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d5bb5aff69cb4e1cb1c5d0d3b55bf27a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d3b04cd2b5d4727b6726b37feaa236e in term 1.
I20260619 08:23:25.125497 27207 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5bb5aff69cb4e1cb1c5d0d3b55bf27a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 36787 } } peers { permanent_uuid: "d5bb5aff69cb4e1cb1c5d0d3b55bf27a" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 39933 } } peers { permanent_uuid: "f031e664ecd7401fa8575f53e2f6a2cd" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } } }
I20260619 08:23:25.125669 27207 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5bb5aff69cb4e1cb1c5d0d3b55bf27a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:23:25.125749 27207 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5bb5aff69cb4e1cb1c5d0d3b55bf27a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d3b04cd2b5d4727b6726b37feaa236e. Latest consensus state: current_term: 1 leader_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 36787 } } peers { permanent_uuid: "d5bb5aff69cb4e1cb1c5d0d3b55bf27a" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 39933 } } peers { permanent_uuid: "f031e664ecd7401fa8575f53e2f6a2cd" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } } }
I20260619 08:23:25.125815 27207 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5bb5aff69cb4e1cb1c5d0d3b55bf27a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:23:25.125846 27207 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5bb5aff69cb4e1cb1c5d0d3b55bf27a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 36787 } } peers { permanent_uuid: "d5bb5aff69cb4e1cb1c5d0d3b55bf27a" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 39933 } } peers { permanent_uuid: "f031e664ecd7401fa8575f53e2f6a2cd" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } } }
I20260619 08:23:25.125895 27207 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5bb5aff69cb4e1cb1c5d0d3b55bf27a [sys.catalog]: This master's current role is: FOLLOWER
W20260619 08:23:25.126341 27218 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d5bb5aff69cb4e1cb1c5d0d3b55bf27a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:23:25.126406 27218 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:23:25.134399 27220 mvcc.cc:205] Tried to move back new op lower bound from 7298487931280158720 to 7298487931152621568. Current Snapshot: MvccSnapshot[applied={T|T < 7298487931202818048}]
W20260619 08:23:25.149024 27009 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:25.147318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57388 (local address 127.4.151.190:36787)
0619 08:23:25.147366 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:25.147368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:25.148287 (+ 919us) server_negotiation.cc:407] Connection header received
0619 08:23:25.148397 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:25.148398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:25.148410 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:25.148441 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:25.148942 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57388: BlockingRecv error: recv got EOF from 127.0.0.1:57388 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260619 08:23:25.149816 27159 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:25.147685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54040 (local address 127.4.151.188:33847)
0619 08:23:25.147727 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:25.147729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:25.149282 (+ 1553us) server_negotiation.cc:407] Connection header received
0619 08:23:25.149405 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:25.149406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:25.149417 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:25.149448 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:25.149755 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54040: BlockingRecv error: recv got EOF from 127.0.0.1:54040 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 08:23:25.150547 27085 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:25.147995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35410 (local address 127.4.151.189:39933)
0619 08:23:25.148035 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:25.148037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:25.150043 (+ 2006us) server_negotiation.cc:407] Connection header received
0619 08:23:25.150139 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:25.150140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:25.150151 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:25.150180 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:25.150486 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35410: BlockingRecv error: recv got EOF from 127.0.0.1:35410 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 08:23:25.152578 27085 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:25.151805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35416 (local address 127.4.151.189:39933)
0619 08:23:25.151851 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:25.151853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:25.152056 (+ 203us) server_negotiation.cc:407] Connection header received
0619 08:23:25.152163 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:25.152164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:25.152182 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:25.152223 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:25.152531 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35416: BlockingRecv error: recv got EOF from 127.0.0.1:35416 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 08:23:25.155778 27085 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:25.154963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35420 (local address 127.4.151.189:39933)
0619 08:23:25.154997 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:25.154999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:25.155245 (+ 246us) server_negotiation.cc:407] Connection header received
0619 08:23:25.155352 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:25.155353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:25.155363 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:25.155397 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:25.155729 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35420: BlockingRecv error: recv got EOF from 127.0.0.1:35420 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 08:23:25.160142 27085 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:25.159197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35434 (local address 127.4.151.189:39933)
0619 08:23:25.159239 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:25.159241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:25.159497 (+ 256us) server_negotiation.cc:407] Connection header received
0619 08:23:25.159608 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:25.159610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:25.159622 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:25.159655 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:25.160085 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35434: BlockingRecv error: recv got EOF from 127.0.0.1:35434 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 08:23:25.166409 27085 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:25.165518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35448 (local address 127.4.151.189:39933)
0619 08:23:25.165558 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:25.165560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:25.165838 (+ 278us) server_negotiation.cc:407] Connection header received
0619 08:23:25.165946 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:25.165947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:25.165959 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:25.165991 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:25.166351 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35448: BlockingRecv error: recv got EOF from 127.0.0.1:35448 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 08:23:25.176646 27085 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:25.175741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35458 (local address 127.4.151.189:39933)
0619 08:23:25.175786 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:25.175788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:25.176042 (+ 254us) server_negotiation.cc:407] Connection header received
0619 08:23:25.176148 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:25.176149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:25.176161 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:25.176192 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:25.176572 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35458: BlockingRecv error: recv got EOF from 127.0.0.1:35458 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 08:23:25.195199 27085 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:25.194165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35464 (local address 127.4.151.189:39933)
0619 08:23:25.194214 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:25.194217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:25.194530 (+ 313us) server_negotiation.cc:407] Connection header received
0619 08:23:25.194652 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:25.194654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:25.194667 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:25.194714 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:25.195134 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35464: BlockingRecv error: recv got EOF from 127.0.0.1:35464 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260619 08:23:25.229993 27085 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:25.228748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35466 (local address 127.4.151.189:39933)
0619 08:23:25.228799 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:25.228801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:25.229151 (+ 350us) server_negotiation.cc:407] Connection header received
0619 08:23:25.229279 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:25.229281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:25.229299 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:25.229334 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:25.229921 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35466: BlockingRecv error: recv got EOF from 127.0.0.1:35466 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260619 08:23:25.296157 27085 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:25.295420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35472 (local address 127.4.151.189:39933)
0619 08:23:25.295489 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:25.295495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:25.295644 (+ 149us) server_negotiation.cc:407] Connection header received
0619 08:23:25.295685 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:25.295686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:25.295700 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:25.295765 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:25.296092 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35472: BlockingRecv error: recv got EOF from 127.0.0.1:35472 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 08:23:25.426149 27085 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:25.425365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35488 (local address 127.4.151.189:39933)
0619 08:23:25.425478 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:25.425480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:25.425492 (+ 12us) server_negotiation.cc:407] Connection header received
0619 08:23:25.425550 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:25.425551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:25.425569 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:25.425644 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:25.426061 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35488: BlockingRecv error: recv got EOF from 127.0.0.1:35488 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 08:23:25.593892 27163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d3b04cd2b5d4727b6726b37feaa236e" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 36787 } } peers { permanent_uuid: "d5bb5aff69cb4e1cb1c5d0d3b55bf27a" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 39933 } } peers { permanent_uuid: "f031e664ecd7401fa8575f53e2f6a2cd" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 33847 } } }
I20260619 08:23:25.594046 27163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d3b04cd2b5d4727b6726b37feaa236e [sys.catalog]: This master's current role is: LEADER
W20260619 08:23:25.684159 27085 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:25.683508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35494 (local address 127.4.151.189:39933)
0619 08:23:25.683586 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:25.683588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:25.683617 (+ 29us) server_negotiation.cc:407] Connection header received
0619 08:23:25.683687 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:25.683688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:25.683699 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:25.683742 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:25.684099 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35494: BlockingRecv error: recv got EOF from 127.0.0.1:35494 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 08:23:26.092919 27198 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f031e664ecd7401fa8575f53e2f6a2cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 08:23:26.126897 27218 catalog_manager.cc:1259] Loaded cluster ID: 5e846cfef20e4d6f9bf4c7599dc730b0
I20260619 08:23:26.127041 27218 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d5bb5aff69cb4e1cb1c5d0d3b55bf27a: loading cluster ID for follower catalog manager: success
I20260619 08:23:26.127939 27218 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d5bb5aff69cb4e1cb1c5d0d3b55bf27a: acquiring CA information for follower catalog manager: success
I20260619 08:23:26.128257 27218 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d5bb5aff69cb4e1cb1c5d0d3b55bf27a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 08:23:26.200326 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:26.197427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35506 (local address 127.4.151.189:39933)
0619 08:23:26.197566 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:26.197568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:26.197645 (+ 77us) server_negotiation.cc:407] Connection header received
0619 08:23:26.197751 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:26.197752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:26.197831 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:26.197921 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:23:26.198076 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:26.198586 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:26.199464 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:26.199687 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:26.199766 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:26.199940 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:23:26.200062 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:23:26.200155 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:26.200204 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 08:23:26.200226 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260619 08:23:26.204859 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:26.202435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54044 (local address 127.4.151.188:33847)
0619 08:23:26.202555 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:26.202556 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:26.202564 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:23:26.202598 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:26.202599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:26.202632 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:26.202699 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:23:26.202820 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:26.203427 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:26.204119 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:26.204334 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:26.204389 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:26.204524 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:23:26.204617 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:23:26.204675 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:26.204714 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 08:23:26.204734 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":71,"threads_started":1}
I20260619 08:23:26.209471 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:26.206923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54048 (local address 127.4.151.188:33847)
0619 08:23:26.206985 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:26.206987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:26.207052 (+ 65us) server_negotiation.cc:407] Connection header received
0619 08:23:26.207124 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:26.207125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:26.207135 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:26.207279 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:23:26.207425 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:26.207898 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:26.208714 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:26.208985 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:26.209063 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:26.209210 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:23:26.209242 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:23:26.209288 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:26.209341 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 08:23:26.209360 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 08:23:26.209759 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:26.206568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35508 (local address 127.4.151.189:39933)
0619 08:23:26.206631 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:26.206633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:26.206684 (+ 51us) server_negotiation.cc:407] Connection header received
0619 08:23:26.206791 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:26.206792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:26.206804 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:26.206862 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:23:26.207075 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:26.207827 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:26.208891 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:26.209220 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:26.209282 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:26.209495 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:23:26.209528 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:23:26.209557 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:26.209656 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 08:23:26.209689 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 08:23:26.210355 27303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:26.206576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57396 (local address 127.4.151.190:36787)
0619 08:23:26.206692 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:26.206694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:26.206941 (+ 247us) server_negotiation.cc:407] Connection header received
0619 08:23:26.206998 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:26.206999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:26.207051 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:26.207128 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:23:26.207349 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:26.208097 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:26.209115 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:26.209499 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:26.209561 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:26.210005 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:23:26.210102 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:23:26.210184 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:26.210249 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 08:23:26.210271 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":61,"threads_started":1}
I20260619 08:23:26.212242 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 26938
I20260619 08:23:26.217715 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 27013
I20260619 08:23:26.223045 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 27088
2026-06-19T08:23:26Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2834 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:23:26 dist-test-slave-d3n8 krb5kdc[27308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:23:26 dist-test-slave-d3n8 krb5kdc[27308](info): set up 2 sockets
Jun 19 08:23:26 dist-test-slave-d3n8 krb5kdc[27308](info): commencing operation
krb5kdc: starting...
W20260619 08:23:27.273861 4702 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/krb5kdc/test-user.keytab.
Jun 19 08:23:27 dist-test-slave-d3n8 krb5kdc[27308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:23:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:23:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.151.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
Entry for principal kudu/127.4.151.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab.
I20260619 08:23:27.400957 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:38627
--webserver_interface=127.4.151.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:41769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:38627,127.4.151.189:43449,127.4.151.188:36033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:23:27.482702 27321 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 08:23:27.482920 27321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:23:27.482937 27321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:23:27.482959 27321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:23:27.484036 27321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:23:27.484077 27321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:23:27.484093 27321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:23:27.484105 27321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:23:27.484117 27321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:23:27.485656 27321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:41769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:38627,127.4.151.189:43449,127.4.151.188:36033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.190.keytab
--principal=kudu/127.4.151.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.190:38627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.151.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:23:27.485849 27321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:23:27.486025 27321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:23:27.490890 27329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:27.490983 27327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:27.490893 27326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:27.491989 27321 server_base.cc:1034] running on GCE node
I20260619 08:23:27.492247 27321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:23:27.492524 27321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:23:27.493677 27321 hybrid_clock.cc:648] HybridClock initialized: now 1781857407493649 us; error 49 us; skew 500 ppm
Jun 19 08:23:27 dist-test-slave-d3n8 krb5kdc[27308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857407, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:23:27.496134 27321 init.cc:377] Logged in from keytab as kudu/127.4.151.190@KRBTEST.COM (short username kudu)
I20260619 08:23:27.496596 27321 webserver.cc:460] Webserver started at http://127.4.151.190:34981/ using document root <none> and password file <none>
I20260619 08:23:27.496824 27321 fs_manager.cc:359] Metadata directory not provided
I20260619 08:23:27.496886 27321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:23:27.497030 27321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:23:27.498143 27321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-0/data/instance:
uuid: "77c844c1883d416d80df3609e1dd7ea6"
format_stamp: "Formatted at 2026-06-19 08:23:27 on dist-test-slave-d3n8"
server_key: "29e58a7383bf0a2fd834d19ae8603400"
server_key_iv: "9effebdbff0af90231475a5cc2418ec3"
server_key_version: "encryptionkey@0"
I20260619 08:23:27.498494 27321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-0/wal/instance:
uuid: "77c844c1883d416d80df3609e1dd7ea6"
format_stamp: "Formatted at 2026-06-19 08:23:27 on dist-test-slave-d3n8"
server_key: "29e58a7383bf0a2fd834d19ae8603400"
server_key_iv: "9effebdbff0af90231475a5cc2418ec3"
server_key_version: "encryptionkey@0"
I20260619 08:23:27.499881 27321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:23:27.500655 27336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:23:27.500828 27321 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:23:27.500926 27321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-0/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-0/wal
uuid: "77c844c1883d416d80df3609e1dd7ea6"
format_stamp: "Formatted at 2026-06-19 08:23:27 on dist-test-slave-d3n8"
server_key: "29e58a7383bf0a2fd834d19ae8603400"
server_key_iv: "9effebdbff0af90231475a5cc2418ec3"
server_key_version: "encryptionkey@0"
I20260619 08:23:27.500999 27321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:23:27.525741 27321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:23:27.527896 27321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:23:27.528064 27321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:23:27.532111 27321 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.190:38627
I20260619 08:23:27.532150 27388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.190:38627 every 8 connection(s)
I20260619 08:23:27.532518 27321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-0/data/info.pb
I20260619 08:23:27.533596 27389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:23:27.535111 27389 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38627 } has no permanent_uuid. Determining permanent_uuid...
I20260619 08:23:27.536139 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 27321
I20260619 08:23:27.536248 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-0/wal/instance
I20260619 08:23:27.536433 4702 external_mini_cluster.cc:1428] Setting key 03cfa059a9952005f21efbb0c24a1e2a
Jun 19 08:23:27 dist-test-slave-d3n8 krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857407, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.151.189@KRBTEST.COM; defaulting to no policy
I20260619 08:23:27.545895 27391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.535766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:38627 (local address 127.0.0.1:60706)
0619 08:23:27.536026 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:27.536073 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 08:23:27.536735 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:23:27.537016 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:23:27.537019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:23:27.537299 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:23:27.537513 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:27.537518 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.538535 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.538536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:27.539208 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:27.539210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.539736 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.539737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:27.539799 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:27.540351 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:23:27.540359 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:23:27.542193 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:23:27.544032 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:27.544041 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:27.544044 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:27.544321 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:27.544646 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:27.544647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:27.544647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:27.544745 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:27.545320 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:23:27.545327 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:23:27.545449 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 08:23:27.545542 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 08:23:27.545692 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":117,"threads_started":1}
I20260619 08:23:27.545923 27392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.535846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60706 (local address 127.4.151.190:38627)
0619 08:23:27.536038 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:27.536048 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:27.536258 (+ 210us) server_negotiation.cc:407] Connection header received
0619 08:23:27.536890 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:27.536892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:27.536912 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:27.537012 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:27.537604 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.538423 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.539308 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.539622 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.539750 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:27.542315 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:27.542323 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:27.542323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:27.542329 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:27.543930 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.544474 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.544477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.544477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.544574 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.544806 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.544806 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.544806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.545149 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:27.545220 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:27.545584 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 08:23:27.545753 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":149,"thread_start_us":70,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.189@KRBTEST.COM" created.
I20260619 08:23:27.546663 27389 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.189" port: 43449 } has no permanent_uuid. Determining permanent_uuid...
I20260619 08:23:27.547030 27391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.546892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:43449 (local address 127.0.0.1:53300)
0619 08:23:27.546945 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:27.547000 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.189:43449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 08:23:27.547161 27338 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.151.189:43449: connect: Connection refused (error 111)
I20260619 08:23:27.547646 27391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.547538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:43449 (local address 127.0.0.1:53314)
0619 08:23:27.547595 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:27.547622 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.189:43449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 08:23:27.547746 27389 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.151.189:43449: Network error: Client connection negotiation failed: client connection to 127.4.151.189:43449: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.189.keytab.
Entry for principal kudu/127.4.151.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.189.keytab.
I20260619 08:23:27.562625 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.189:43449
--webserver_interface=127.4.151.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.189.keytab
--principal=kudu/127.4.151.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:41769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:38627,127.4.151.189:43449,127.4.151.188:36033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:23:27.596987 27389 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.151.189" port: 43449 } attempt: 1
I20260619 08:23:27.597579 27391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.597435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:43449 (local address 127.0.0.1:53324)
0619 08:23:27.597494 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:27.597542 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.189:43449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 08:23:27.597939 27391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.597849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:43449 (local address 127.0.0.1:53340)
0619 08:23:27.597891 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:27.597919 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.189:43449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 08:23:27.598033 27389 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.151.189:43449: Network error: Client connection negotiation failed: client connection to 127.4.151.189:43449: connect: Connection refused (error 111)
W20260619 08:23:27.644086 27396 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 08:23:27.644320 27396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:23:27.644336 27396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:23:27.644378 27396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:23:27.645553 27396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:23:27.645592 27396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:23:27.645606 27396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:23:27.645619 27396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:23:27.645664 27396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:23:27.647228 27396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:41769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:38627,127.4.151.189:43449,127.4.151.188:36033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.189.keytab
--principal=kudu/127.4.151.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.189:43449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.151.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:23:27.647485 27396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:23:27.647711 27396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:23:27.652549 27404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:27.652549 27401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:27.652570 27402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:27.652755 27396 server_base.cc:1034] running on GCE node
I20260619 08:23:27.653065 27396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:23:27.653364 27396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:23:27.654578 27396 hybrid_clock.cc:648] HybridClock initialized: now 1781857407654560 us; error 29 us; skew 500 ppm
Jun 19 08:23:27 dist-test-slave-d3n8 krb5kdc[27308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857407, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:23:27.657231 27396 init.cc:377] Logged in from keytab as kudu/127.4.151.189@KRBTEST.COM (short username kudu)
I20260619 08:23:27.657711 27396 webserver.cc:460] Webserver started at http://127.4.151.189:42495/ using document root <none> and password file <none>
I20260619 08:23:27.657902 27396 fs_manager.cc:359] Metadata directory not provided
I20260619 08:23:27.657940 27396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:23:27.658027 27396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:23:27.659029 27396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-1/data/instance:
uuid: "f9478c21d299492fb7f6b216d2e42d97"
format_stamp: "Formatted at 2026-06-19 08:23:27 on dist-test-slave-d3n8"
server_key: "bc1f8836b58639c21291eb80643a912a"
server_key_iv: "608d3dcbf2951fff43e8279232d68856"
server_key_version: "encryptionkey@0"
I20260619 08:23:27.659341 27396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-1/wal/instance:
uuid: "f9478c21d299492fb7f6b216d2e42d97"
format_stamp: "Formatted at 2026-06-19 08:23:27 on dist-test-slave-d3n8"
server_key: "bc1f8836b58639c21291eb80643a912a"
server_key_iv: "608d3dcbf2951fff43e8279232d68856"
server_key_version: "encryptionkey@0"
I20260619 08:23:27.660570 27396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:23:27.661314 27411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:23:27.661494 27396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:23:27.661583 27396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-1/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-1/wal
uuid: "f9478c21d299492fb7f6b216d2e42d97"
format_stamp: "Formatted at 2026-06-19 08:23:27 on dist-test-slave-d3n8"
server_key: "bc1f8836b58639c21291eb80643a912a"
server_key_iv: "608d3dcbf2951fff43e8279232d68856"
server_key_version: "encryptionkey@0"
I20260619 08:23:27.661657 27396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:23:27.666170 27389 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.151.189" port: 43449 } attempt: 2
I20260619 08:23:27.666541 27391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.666425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:43449 (local address 127.0.0.1:53352)
0619 08:23:27.666479 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:27.666514 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.189:43449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 08:23:27.666901 27391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.666812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:43449 (local address 127.0.0.1:53368)
0619 08:23:27.666853 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:27.666877 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.189:43449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 08:23:27.667019 27389 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.151.189:43449: Network error: Client connection negotiation failed: client connection to 127.4.151.189:43449: connect: Connection refused (error 111)
I20260619 08:23:27.669588 27396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:23:27.671746 27396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:23:27.671888 27396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:23:27.676043 27396 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.189:43449
I20260619 08:23:27.676087 27463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.189:43449 every 8 connection(s)
I20260619 08:23:27.676415 27396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-1/data/info.pb
I20260619 08:23:27.677421 27464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:23:27.677448 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 27396
I20260619 08:23:27.677534 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-1/wal/instance
I20260619 08:23:27.677742 4702 external_mini_cluster.cc:1428] Setting key 9635a21c9fac13e838bbc1aa4e10bb00
I20260619 08:23:27.679725 27464 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38627 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.151.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.151.188@KRBTEST.COM" created.
Jun 19 08:23:27 dist-test-slave-d3n8 krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857407, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.189@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:23:27.692042 27392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.680424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60720 (local address 127.4.151.190:38627)
0619 08:23:27.680483 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:27.680485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:27.680694 (+ 209us) server_negotiation.cc:407] Connection header received
0619 08:23:27.681275 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:27.681276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:27.681291 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:27.681357 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:27.681984 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.682473 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.683437 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.683644 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.683727 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:27.688702 (+ 4975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:27.688704 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:27.688705 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:27.688714 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:27.690420 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.690995 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.690996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.690997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.691075 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.691372 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.691373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.691374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.691500 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:27.691573 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:27.691847 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 08:23:27.691936 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 08:23:27.692219 27467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.680404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:38627 (local address 127.0.0.1:60720)
0619 08:23:27.680568 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:27.680602 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 08:23:27.681086 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:23:27.681454 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:23:27.681458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:23:27.681706 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:23:27.681894 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:27.681898 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.682564 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.682565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:27.683352 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:27.683354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.683749 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.683750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:27.683876 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:27.684395 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:23:27.684403 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:23:27.688604 (+ 4201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:23:27.690521 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:27.690527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:27.690530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:27.690785 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:27.691147 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:27.691148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:27.691149 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:27.691297 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:27.691565 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:23:27.691571 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:23:27.691718 (+ 147us) client_negotiation.cc:769] Sending connection context
0619 08:23:27.691827 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 08:23:27.691988 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260619 08:23:27.692754 27464 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.189" port: 43449 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 08:23:27 dist-test-slave-d3n8 krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857407, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.189@KRBTEST.COM for kudu/127.4.151.189@KRBTEST.COM
I20260619 08:23:27.702643 27467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.693045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:43449 (local address 127.0.0.1:53372)
0619 08:23:27.693309 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:27.693321 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:23:27.693413 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:23:27.693590 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:23:27.693591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:23:27.693790 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:23:27.693862 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:27.693864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.694755 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.694757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:27.695743 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:27.695746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.696205 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.696206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:27.696296 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:27.696700 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:23:27.696707 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:23:27.699515 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:23:27.701368 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:27.701369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:27.701370 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:27.701673 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:27.701923 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:27.701923 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:27.701924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:27.701982 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:27.702385 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:23:27.702386 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:23:27.702425 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 08:23:27.702477 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 08:23:27.702553 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245}
I20260619 08:23:27.702806 27469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.693134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53372 (local address 127.4.151.189:43449)
0619 08:23:27.693308 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:27.693311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:27.693414 (+ 103us) server_negotiation.cc:407] Connection header received
0619 08:23:27.693489 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:27.693491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:27.693502 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:27.693562 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:27.693944 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.694643 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.695834 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.696126 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.696185 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:27.699654 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:27.699664 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:27.699664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:27.699672 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:27.701290 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.701771 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.701774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.701774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.701837 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.702066 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.702067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.702067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.702316 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:27.702388 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:27.702569 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 08:23:27.702637 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":109,"threads_started":1}
I20260619 08:23:27.703348 27464 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } has no permanent_uuid. Determining permanent_uuid...
I20260619 08:23:27.703667 27467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.703577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:36033 (local address 127.0.0.1:33730)
0619 08:23:27.703607 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:27.703650 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260619 08:23:27.703833 27412 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.151.188:36033: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.151.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.188.keytab.
Entry for principal kudu/127.4.151.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.188.keytab.
I20260619 08:23:27.704270 27467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.704192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:36033 (local address 127.0.0.1:33740)
0619 08:23:27.704226 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:27.704249 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260619 08:23:27.704372 27464 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.151.188:36033: Network error: Client connection negotiation failed: client connection to 127.4.151.188:36033: connect: Connection refused (error 111)
I20260619 08:23:27.704912 4702 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR4iS53/build/release/bin/kudu
/tmp/dist-test-taskR4iS53/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.188:36033
--webserver_interface=127.4.151.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.188.keytab
--principal=kudu/127.4.151.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.151.148:41769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.151.190:38627,127.4.151.189:43449,127.4.151.188:36033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:23:27.753587 27464 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } attempt: 1
I20260619 08:23:27.754096 27467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.753954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:36033 (local address 127.0.0.1:33744)
0619 08:23:27.754017 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:27.754055 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 08:23:27.754431 27467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.754334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:36033 (local address 127.0.0.1:33760)
0619 08:23:27.754384 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:27.754408 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 08:23:27.754508 27464 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.151.188:36033: Network error: Client connection negotiation failed: client connection to 127.4.151.188:36033: connect: Connection refused (error 111)
I20260619 08:23:27.758251 27389 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.151.189" port: 43449 } attempt: 3
Jun 19 08:23:27 dist-test-slave-d3n8 krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857407, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for kudu/127.4.151.189@KRBTEST.COM
I20260619 08:23:27.766575 27391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.758581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:43449 (local address 127.0.0.1:53380)
0619 08:23:27.758636 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:27.758648 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:23:27.758698 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:23:27.758885 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:23:27.758886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:23:27.759030 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:23:27.759111 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:27.759112 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.760081 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.760082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:27.760727 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:27.760729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.761273 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.761274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:27.761348 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:27.761901 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:23:27.761906 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:23:27.763471 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:23:27.765519 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:27.765521 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:27.765521 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:27.765770 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:27.765972 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:27.765973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:27.765973 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:27.766029 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:27.766306 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:23:27.766307 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:23:27.766352 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 08:23:27.766415 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 08:23:27.766500 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 08:23:27.766628 27469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.758669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53380 (local address 127.4.151.189:43449)
0619 08:23:27.758743 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:27.758745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:27.758755 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:23:27.758785 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:27.758786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:27.758799 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:27.758863 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:27.759194 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.759978 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.760828 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.761174 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.761250 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:27.763563 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:27.763564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:27.763565 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:27.763572 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:27.765437 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.765844 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.765845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.765846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.765908 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.766090 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.766091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.766091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.766225 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:27.766291 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:27.766453 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 08:23:27.766534 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 08:23:27.767443 27389 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } has no permanent_uuid. Determining permanent_uuid...
I20260619 08:23:27.767792 27391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.767688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:36033 (local address 127.0.0.1:33776)
0619 08:23:27.767735 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:27.767768 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 08:23:27.768175 27391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.768080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:36033 (local address 127.0.0.1:33784)
0619 08:23:27.768126 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:27.768153 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 08:23:27.768294 27389 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.151.188:36033: Network error: Client connection negotiation failed: client connection to 127.4.151.188:36033: connect: Connection refused (error 111)
W20260619 08:23:27.789624 27471 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 08:23:27.789932 27471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:23:27.789968 27471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:23:27.790011 27471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:23:27.791332 27471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:23:27.791378 27471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:23:27.791391 27471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:23:27.791404 27471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:23:27.791416 27471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:23:27.792847 27471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.151.148:41769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.151.190:38627,127.4.151.189:43449,127.4.151.188:36033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/krb5kdc/kudu_127.4.151.188.keytab
--principal=kudu/127.4.151.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.151.188:36033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.151.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 06:52:04 UTC on ccycloud.quasar-kfejyx.root.comops.site
build id 4455157
I20260619 08:23:27.793049 27471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:23:27.793223 27471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:23:27.798499 27477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:23:27.798493 27479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:27.798818 27471 server_base.cc:1034] running on GCE node
W20260619 08:23:27.798493 27476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:23:27.799276 27471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:23:27.799440 27389 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } attempt: 1
I20260619 08:23:27.799585 27471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:23:27.799849 27391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.799707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:36033 (local address 127.0.0.1:33800)
0619 08:23:27.799763 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:27.799805 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 08:23:27.800225 27391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.800090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:36033 (local address 127.0.0.1:33812)
0619 08:23:27.800149 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:27.800200 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 08:23:27.800329 27389 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.151.188:36033: Network error: Client connection negotiation failed: client connection to 127.4.151.188:36033: connect: Connection refused (error 111)
I20260619 08:23:27.800755 27471 hybrid_clock.cc:648] HybridClock initialized: now 1781857407800729 us; error 42 us; skew 500 ppm
Jun 19 08:23:27 dist-test-slave-d3n8 krb5kdc[27308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857407, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:23:27.803331 27471 init.cc:377] Logged in from keytab as kudu/127.4.151.188@KRBTEST.COM (short username kudu)
I20260619 08:23:27.803860 27471 webserver.cc:460] Webserver started at http://127.4.151.188:34059/ using document root <none> and password file <none>
I20260619 08:23:27.804069 27471 fs_manager.cc:359] Metadata directory not provided
I20260619 08:23:27.804111 27471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:23:27.804201 27471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:23:27.805258 27471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-2/data/instance:
uuid: "85e67a14487f4813aaff79b9be811a29"
format_stamp: "Formatted at 2026-06-19 08:23:27 on dist-test-slave-d3n8"
server_key: "409f0ba06dad1ff03dc97199688c4a02"
server_key_iv: "8c2d998d2436851efa46b27583335e19"
server_key_version: "encryptionkey@0"
I20260619 08:23:27.805627 27471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-2/wal/instance:
uuid: "85e67a14487f4813aaff79b9be811a29"
format_stamp: "Formatted at 2026-06-19 08:23:27 on dist-test-slave-d3n8"
server_key: "409f0ba06dad1ff03dc97199688c4a02"
server_key_iv: "8c2d998d2436851efa46b27583335e19"
server_key_version: "encryptionkey@0"
I20260619 08:23:27.806936 27471 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:23:27.807653 27486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:23:27.807847 27471 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:23:27.807950 27471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-2/data,/tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-2/wal
uuid: "85e67a14487f4813aaff79b9be811a29"
format_stamp: "Formatted at 2026-06-19 08:23:27 on dist-test-slave-d3n8"
server_key: "409f0ba06dad1ff03dc97199688c4a02"
server_key_iv: "8c2d998d2436851efa46b27583335e19"
server_key_version: "encryptionkey@0"
I20260619 08:23:27.808022 27471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:23:27.822657 27464 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } attempt: 2
I20260619 08:23:27.823197 27467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.823057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:36033 (local address 127.0.0.1:33814)
0619 08:23:27.823117 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:27.823164 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 08:23:27.823539 27467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.823463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:36033 (local address 127.0.0.1:33816)
0619 08:23:27.823500 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:27.823524 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.151.188:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 08:23:27.823657 27464 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.151.188:36033: Network error: Client connection negotiation failed: client connection to 127.4.151.188:36033: connect: Connection refused (error 111)
I20260619 08:23:27.839918 27471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:23:27.842236 27471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:23:27.842443 27471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:23:27.847348 27471 rpc_server.cc:307] RPC server started. Bound to: 127.4.151.188:36033
I20260619 08:23:27.847378 27538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.151.188:36033 every 8 connection(s)
I20260619 08:23:27.847766 27471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-2/data/info.pb
I20260619 08:23:27.848865 27539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:23:27.850136 4702 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR4iS53/build/release/bin/kudu as pid 27471
I20260619 08:23:27.850245 4702 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR4iS53/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781857130600961-4702-0/minicluster-data/master-2/wal/instance
I20260619 08:23:27.850467 4702 external_mini_cluster.cc:1428] Setting key 6ab5218a478735da17e35bb342a66028
I20260619 08:23:27.850849 27539 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38627 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 08:23:27 dist-test-slave-d3n8 krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857407, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.188@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
Jun 19 08:23:27 dist-test-slave-d3n8 krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.188@KRBTEST.COM
I20260619 08:23:27.861172 27392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.851575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60734 (local address 127.4.151.190:38627)
0619 08:23:27.851623 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:27.851625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:27.851738 (+ 113us) server_negotiation.cc:407] Connection header received
0619 08:23:27.852128 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:27.852148 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:27.852161 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:27.852248 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:27.852925 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.853611 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.854604 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.854957 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.855016 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:27.857902 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:27.857904 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:27.857905 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:27.857919 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:27.859496 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.859969 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.859971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.859971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.860044 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.860418 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.860418 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.860419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.860537 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:27.860627 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:27.860980 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 08:23:27.861070 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 08:23:27.861352 27543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.851481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.190:38627 (local address 127.0.0.1:60734)
0619 08:23:27.851629 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:27.851673 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 08:23:27.851883 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:23:27.852324 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:23:27.852327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:23:27.852578 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:23:27.852808 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:27.852814 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.853729 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.853730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:27.854506 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:27.854508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.855046 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.855047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:27.855181 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:27.855925 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:23:27.855933 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:23:27.857745 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:23:27.859600 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:27.859606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:27.859608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:27.859879 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:27.860154 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:27.860155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:27.860155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:27.860337 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:27.860624 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:23:27.860629 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:23:27.860835 (+ 206us) client_negotiation.cc:769] Sending connection context
0619 08:23:27.860938 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 08:23:27.861092 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":26,"thread_start_us":85,"threads_started":1}
I20260619 08:23:27.861778 27542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.851075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33822 (local address 127.4.151.188:36033)
0619 08:23:27.851255 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:27.851262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:27.851287 (+ 25us) server_negotiation.cc:407] Connection header received
0619 08:23:27.851881 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:27.851890 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:27.851915 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:27.852001 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:27.852851 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.853848 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.854912 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.855453 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.855509 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:27.858203 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:27.858211 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:27.858211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:27.858217 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:27.860194 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.860592 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.860595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.860596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.860654 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.860855 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.860855 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.860855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.861058 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:27.861197 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:27.861450 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 08:23:27.861539 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":143,"thread_start_us":80,"threads_started":1}
I20260619 08:23:27.861900 27539 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.189" port: 43449 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 08:23:27 dist-test-slave-d3n8 krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857407, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.188@KRBTEST.COM for kudu/127.4.151.189@KRBTEST.COM
Jun 19 08:23:27 dist-test-slave-d3n8 krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.190@KRBTEST.COM
I20260619 08:23:27.869768 27543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.862165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.189:43449 (local address 127.0.0.1:53392)
0619 08:23:27.862223 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:27.862234 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:23:27.862267 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:23:27.862406 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:23:27.862406 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:23:27.862517 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:23:27.862580 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:27.862582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.863531 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.863532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:27.864324 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:27.864327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.864855 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.864856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:27.864924 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:27.865504 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:23:27.865509 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:23:27.866860 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:23:27.868637 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:27.868638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:27.868638 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:27.868847 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:27.869091 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:27.869092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:27.869092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:27.869156 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:27.869493 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:23:27.869495 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:23:27.869546 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 08:23:27.869617 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 08:23:27.869699 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 08:23:27.869885 27469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.862223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53392 (local address 127.4.151.189:43449)
0619 08:23:27.862275 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:27.862277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:27.862287 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:23:27.862321 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:27.862321 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:27.862332 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:27.862394 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:27.862659 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.863449 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.864435 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.864756 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.864826 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:27.866942 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:27.866944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:27.866945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:27.866952 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:27.868546 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.868952 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.868953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.868954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.869014 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.869227 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.869227 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.869228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.869349 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:27.869503 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:27.869668 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 08:23:27.869755 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 08:23:27.870260 27539 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } has no permanent_uuid. Determining permanent_uuid...
I20260619 08:23:27.871065 27392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.862614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60736 (local address 127.4.151.190:38627)
0619 08:23:27.862687 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:27.862689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:27.862696 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:23:27.862719 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:27.862720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:27.862727 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:27.862768 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:27.863369 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.864078 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.865109 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.865417 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.865483 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:27.867815 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:27.867817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:27.867817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:27.867824 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:27.869492 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.869955 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.869957 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.869957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.870018 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.870290 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.870290 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.870291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.870632 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:27.870845 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:27.870887 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 08:23:27.870962 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Jun 19 08:23:27 dist-test-slave-d3n8 krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857407, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.188@KRBTEST.COM for kudu/127.4.151.188@KRBTEST.COM
I20260619 08:23:27.875479 27389 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } attempt: 2
Jun 19 08:23:27 dist-test-slave-d3n8 krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.151.189@KRBTEST.COM
I20260619 08:23:27.878190 27543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.870518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:36033 (local address 127.0.0.1:33826)
0619 08:23:27.870566 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:27.870576 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:23:27.870613 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:23:27.870762 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:23:27.870762 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:23:27.870903 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:23:27.870943 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:27.870945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.871979 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.871980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:27.872504 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:27.872506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.872910 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.872911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:27.872964 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:27.873467 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:23:27.873472 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:23:27.874823 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:23:27.876737 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:27.876738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:27.876738 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:27.876977 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:27.877496 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:27.877496 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:27.877497 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:27.877563 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:27.877878 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:23:27.877878 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:23:27.877918 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 08:23:27.877988 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 08:23:27.878098 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 08:23:27.878291 27542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.870566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33826 (local address 127.4.151.188:36033)
0619 08:23:27.870623 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:27.870625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:27.870633 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:23:27.870656 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:27.870657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:27.870682 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:27.870741 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:27.871126 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.871883 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.872598 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.872823 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.872910 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:27.874919 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:27.874920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:27.874920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:27.874927 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:27.876659 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.877328 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.877329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.877330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.877419 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.877637 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.877638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.877638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.877800 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:27.877863 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:27.878031 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 08:23:27.878176 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":37}
I20260619 08:23:27.880170 27469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.871691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53404 (local address 127.4.151.189:43449)
0619 08:23:27.871730 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:27.871732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:27.871773 (+ 41us) server_negotiation.cc:407] Connection header received
0619 08:23:27.871835 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:27.871836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:27.871845 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:27.871900 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:27.872365 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.872815 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.873479 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.873680 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.873756 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:27.876815 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:27.876817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:27.876817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:27.876824 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:27.878627 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.879056 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.879057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.879058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.879119 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.879382 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.879383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.879383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.879531 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:27.879689 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:27.880049 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 08:23:27.880108 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:23:27.880718 27539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85e67a14487f4813aaff79b9be811a29: Bootstrap starting.
Jun 19 08:23:27 dist-test-slave-d3n8 krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857407, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.190@KRBTEST.COM for kudu/127.4.151.188@KRBTEST.COM
I20260619 08:23:27.881518 27539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85e67a14487f4813aaff79b9be811a29: Neither blocks nor log segments found. Creating new log.
I20260619 08:23:27.881915 27539 log.cc:824] T 00000000000000000000000000000000 P 85e67a14487f4813aaff79b9be811a29: Log is configured to *not* fsync() on all Append() calls
I20260619 08:23:27.882820 27539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85e67a14487f4813aaff79b9be811a29: No bootstrap required, opened a new log
I20260619 08:23:27.883466 4702 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:23:27.884738 27391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.876297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:36033 (local address 127.0.0.1:33834)
0619 08:23:27.876357 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:27.876370 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:23:27.876415 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:23:27.876797 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:23:27.876799 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:23:27.876960 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:23:27.877011 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:27.877013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.878055 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.878056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:27.878881 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:27.878884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.879507 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.879508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:27.879571 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:27.879972 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:23:27.879978 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:23:27.881254 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:23:27.883446 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:27.883447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:27.883448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:27.883693 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:27.883953 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:27.883955 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:27.883955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:27.884034 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:27.884374 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:23:27.884375 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:23:27.884423 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 08:23:27.884510 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 08:23:27.884632 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 08:23:27.885214 27539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85e67a14487f4813aaff79b9be811a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c844c1883d416d80df3609e1dd7ea6" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38627 } } peers { permanent_uuid: "f9478c21d299492fb7f6b216d2e42d97" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 43449 } } peers { permanent_uuid: "85e67a14487f4813aaff79b9be811a29" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } }
I20260619 08:23:27.885411 27539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85e67a14487f4813aaff79b9be811a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e67a14487f4813aaff79b9be811a29, State: Initialized, Role: FOLLOWER
I20260619 08:23:27.885509 27539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85e67a14487f4813aaff79b9be811a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c844c1883d416d80df3609e1dd7ea6" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38627 } } peers { permanent_uuid: "f9478c21d299492fb7f6b216d2e42d97" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 43449 } } peers { permanent_uuid: "85e67a14487f4813aaff79b9be811a29" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } }
I20260619 08:23:27.885792 27546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e67a14487f4813aaff79b9be811a29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c844c1883d416d80df3609e1dd7ea6" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38627 } } peers { permanent_uuid: "f9478c21d299492fb7f6b216d2e42d97" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 43449 } } peers { permanent_uuid: "85e67a14487f4813aaff79b9be811a29" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } } }
I20260619 08:23:27.885877 27539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85e67a14487f4813aaff79b9be811a29 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:23:27.884783 27363 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "77c844c1883d416d80df3609e1dd7ea6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60736
I20260619 08:23:27.885890 27546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e67a14487f4813aaff79b9be811a29 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:23:27.886238 27544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.876484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33834 (local address 127.4.151.188:36033)
0619 08:23:27.876638 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:27.876640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:27.876646 (+ 6us) server_negotiation.cc:407] Connection header received
0619 08:23:27.876664 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:27.876665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:27.876680 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:27.877084 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:27.877126 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.877967 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.879152 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.879429 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.879495 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:27.881447 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:27.881448 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:27.881449 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:27.881456 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:27.883336 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.883788 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.883789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.883789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.883875 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.884113 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.884114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.884114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.884293 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:27.884377 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:27.884556 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 08:23:27.884647 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":108,"threads_started":1}
I20260619 08:23:27.887873 27389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6: Bootstrap starting.
I20260619 08:23:27.888614 27389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6: Neither blocks nor log segments found. Creating new log.
I20260619 08:23:27.888914 27389 log.cc:824] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6: Log is configured to *not* fsync() on all Append() calls
W20260619 08:23:27.888088 27557 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85e67a14487f4813aaff79b9be811a29: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:23:27.889509 27557 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:23:27.889810 27389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6: No bootstrap required, opened a new log
I20260619 08:23:27.891327 27389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c844c1883d416d80df3609e1dd7ea6" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38627 } } peers { permanent_uuid: "f9478c21d299492fb7f6b216d2e42d97" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 43449 } } peers { permanent_uuid: "85e67a14487f4813aaff79b9be811a29" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } }
I20260619 08:23:27.891469 27389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c844c1883d416d80df3609e1dd7ea6, State: Initialized, Role: FOLLOWER
I20260619 08:23:27.891546 27389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c844c1883d416d80df3609e1dd7ea6" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38627 } } peers { permanent_uuid: "f9478c21d299492fb7f6b216d2e42d97" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 43449 } } peers { permanent_uuid: "85e67a14487f4813aaff79b9be811a29" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } }
I20260619 08:23:27.891876 27389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:23:27.891844 27560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c844c1883d416d80df3609e1dd7ea6" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38627 } } peers { permanent_uuid: "f9478c21d299492fb7f6b216d2e42d97" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 43449 } } peers { permanent_uuid: "85e67a14487f4813aaff79b9be811a29" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } } }
I20260619 08:23:27.891929 27560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:23:27.890095 27471 master_runner.cc:186] Error getting master registration for 127.4.151.190:38627: OK, instance_id { permanent_uuid: "77c844c1883d416d80df3609e1dd7ea6" instance_seqno: 1781857407530992 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 08:23:27.892880 27471 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 08:23:27.892997 27363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 08:23:27.893065 27363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:23:27.894199 27363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c844c1883d416d80df3609e1dd7ea6" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38627 } } peers { permanent_uuid: "f9478c21d299492fb7f6b216d2e42d97" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 43449 } } peers { permanent_uuid: "85e67a14487f4813aaff79b9be811a29" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } }
I20260619 08:23:27.894654 27363 leader_election.cc:288] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6 [CANDIDATE]: Term 1 election: Requested vote from peers f9478c21d299492fb7f6b216d2e42d97 (127.4.151.189:43449), 85e67a14487f4813aaff79b9be811a29 (127.4.151.188:36033)
I20260619 08:23:27.895143 27513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77c844c1883d416d80df3609e1dd7ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "85e67a14487f4813aaff79b9be811a29"
I20260619 08:23:27.895273 27513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85e67a14487f4813aaff79b9be811a29 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:23:27.896281 27513 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 85e67a14487f4813aaff79b9be811a29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77c844c1883d416d80df3609e1dd7ea6 in term 1.
I20260619 08:23:27.896518 27337 leader_election.cc:302] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77c844c1883d416d80df3609e1dd7ea6, 85e67a14487f4813aaff79b9be811a29; no voters:
I20260619 08:23:27.896638 27560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:23:27.896785 27560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 77c844c1883d416d80df3609e1dd7ea6, State: Running, Role: LEADER
I20260619 08:23:27.896873 27560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c844c1883d416d80df3609e1dd7ea6" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38627 } } peers { permanent_uuid: "f9478c21d299492fb7f6b216d2e42d97" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 43449 } } peers { permanent_uuid: "85e67a14487f4813aaff79b9be811a29" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } }
I20260619 08:23:27.897555 27579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77c844c1883d416d80df3609e1dd7ea6. Latest consensus state: current_term: 1 leader_uuid: "77c844c1883d416d80df3609e1dd7ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c844c1883d416d80df3609e1dd7ea6" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38627 } } peers { permanent_uuid: "f9478c21d299492fb7f6b216d2e42d97" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 43449 } } peers { permanent_uuid: "85e67a14487f4813aaff79b9be811a29" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } } }
I20260619 08:23:27.897637 27579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6 [sys.catalog]: This master's current role is: LEADER
W20260619 08:23:27.898023 27571 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:23:27.898116 27571 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:23:27.896745 27321 master_runner.cc:186] Error getting master registration for 127.4.151.189:43449: OK, instance_id { permanent_uuid: "f9478c21d299492fb7f6b216d2e42d97" instance_seqno: 1781857407674915 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 08:23:27.899192 27321 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 08:23:27.906142 27392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.895900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60748 (local address 127.4.151.190:38627)
0619 08:23:27.895944 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:27.895946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:27.898332 (+ 2386us) server_negotiation.cc:407] Connection header received
0619 08:23:27.898433 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:27.898434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:27.898444 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:27.898473 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:27.899363 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.900113 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.901217 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.901568 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.901611 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:27.903238 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:27.903239 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:27.903239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:27.903246 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:27.904916 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.905307 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.905308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.905309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.905360 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.905579 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.905580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.905580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.905707 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:27.905776 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:27.905972 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 08:23:27.906047 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:23:27.910519 27581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:23:27.910773 27581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:23:27.914144 27544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.896522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33846 (local address 127.4.151.188:36033)
0619 08:23:27.896556 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:27.896558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:27.899517 (+ 2959us) server_negotiation.cc:407] Connection header received
0619 08:23:27.899799 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:27.899800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:27.899824 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:27.899854 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:27.902917 (+ 3063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.903690 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.906937 (+ 3247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.907389 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.907458 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:27.909562 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:27.909563 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:27.909563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:27.909571 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:27.912002 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.913205 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.913206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.913206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.913267 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.913485 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.913485 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.913486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.913620 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:27.913738 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:27.913935 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 08:23:27.914028 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:23:27.914929 27469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.896230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53408 (local address 127.4.151.189:43449)
0619 08:23:27.896278 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:27.896280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:27.899592 (+ 3312us) server_negotiation.cc:407] Connection header received
0619 08:23:27.899944 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:27.899945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:27.899957 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:27.899989 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:27.902045 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.902784 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.907544 (+ 4760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.908157 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.908220 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:27.911333 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:27.911335 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:27.911335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:27.911343 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:27.912966 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.913573 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.913574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.913575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.913646 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.913959 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.913960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.913960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.914118 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:27.914197 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:27.914741 (+ 544us) server_negotiation.cc:299] Negotiation successful
0619 08:23:27.914836 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:23:27.914995 27464 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } attempt: 3
I20260619 08:23:27.916060 27513 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85e67a14487f4813aaff79b9be811a29 [term 1 FOLLOWER]: Refusing update from remote peer 77c844c1883d416d80df3609e1dd7ea6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:23:27.916492 27579 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e67a14487f4813aaff79b9be811a29" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:23:27.919361 27546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e67a14487f4813aaff79b9be811a29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77c844c1883d416d80df3609e1dd7ea6. Latest consensus state: current_term: 1 leader_uuid: "77c844c1883d416d80df3609e1dd7ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c844c1883d416d80df3609e1dd7ea6" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38627 } } peers { permanent_uuid: "f9478c21d299492fb7f6b216d2e42d97" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 43449 } } peers { permanent_uuid: "85e67a14487f4813aaff79b9be811a29" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } } }
I20260619 08:23:27.919454 27546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e67a14487f4813aaff79b9be811a29 [sys.catalog]: This master's current role is: FOLLOWER
Jun 19 08:23:27 dist-test-slave-d3n8 krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781857407, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.151.189@KRBTEST.COM for kudu/127.4.151.188@KRBTEST.COM
I20260619 08:23:27.925211 27560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "77c844c1883d416d80df3609e1dd7ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c844c1883d416d80df3609e1dd7ea6" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38627 } } peers { permanent_uuid: "f9478c21d299492fb7f6b216d2e42d97" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 43449 } } peers { permanent_uuid: "85e67a14487f4813aaff79b9be811a29" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } } }
I20260619 08:23:27.925328 27560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6 [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:27.925292 27546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e67a14487f4813aaff79b9be811a29 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "77c844c1883d416d80df3609e1dd7ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c844c1883d416d80df3609e1dd7ea6" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38627 } } peers { permanent_uuid: "f9478c21d299492fb7f6b216d2e42d97" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 43449 } } peers { permanent_uuid: "85e67a14487f4813aaff79b9be811a29" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } } }
I20260619 08:23:27.925360 27546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e67a14487f4813aaff79b9be811a29 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:23:27.925427 27583 mvcc.cc:205] Tried to move back new op lower bound from 7298487942821298176 to 7298487942746177536. Current Snapshot: MvccSnapshot[applied={T|T < 7298487942821298176}]
I20260619 08:23:27.926092 27467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.915216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.151.188:36033 (local address 127.0.0.1:33856)
0619 08:23:27.915311 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 08:23:27.915321 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:23:27.915357 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:23:27.915590 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:23:27.915591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:23:27.915750 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:23:27.915794 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:27.915797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.916662 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.916663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:27.917472 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:23:27.917474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.919516 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.919518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:23:27.919583 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:27.920023 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:23:27.920028 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:23:27.921896 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:23:27.924295 (+ 2399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:27.924296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:27.924297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:27.924508 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:27.924881 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:23:27.924882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:23:27.924882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:23:27.924962 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:23:27.925778 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:23:27.925779 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:23:27.925826 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 08:23:27.925891 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 08:23:27.925990 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 08:23:27.926224 27581 catalog_manager.cc:1347] Generated new cluster ID: 501bb741dc584839ad8e7648ba371b55
I20260619 08:23:27.926271 27581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:23:27.933667 27544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:27.915424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33856 (local address 127.4.151.188:36033)
0619 08:23:27.915459 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:27.915461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:27.915467 (+ 6us) server_negotiation.cc:407] Connection header received
0619 08:23:27.915487 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:27.915488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:27.915497 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:27.915546 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:27.915878 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.916571 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.917548 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:27.917856 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:27.917924 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:27.921986 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:23:27.921987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:23:27.921987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:23:27.921994 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:23:27.923772 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.924608 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.924609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.924609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.924687 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:23:27.925536 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:23:27.925537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:23:27.925537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:23:27.925674 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:23:27.925748 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:27.933462 (+ 7714us) server_negotiation.cc:299] Negotiation successful
0619 08:23:27.933568 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 08:23:27.935312 27464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9478c21d299492fb7f6b216d2e42d97: Bootstrap starting.
I20260619 08:23:27.936056 27464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9478c21d299492fb7f6b216d2e42d97: Neither blocks nor log segments found. Creating new log.
I20260619 08:23:27.936347 27464 log.cc:824] T 00000000000000000000000000000000 P f9478c21d299492fb7f6b216d2e42d97: Log is configured to *not* fsync() on all Append() calls
I20260619 08:23:27.937978 27584 mvcc.cc:205] Tried to move back new op lower bound from 7298487942821298176 to 7298487942746177536. Current Snapshot: MvccSnapshot[applied={T|T < 7298487942821298176}]
I20260619 08:23:27.941129 27581 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:23:27.941877 27581 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:23:27.942063 27464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9478c21d299492fb7f6b216d2e42d97: No bootstrap required, opened a new log
I20260619 08:23:27.944098 27464 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9478c21d299492fb7f6b216d2e42d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c844c1883d416d80df3609e1dd7ea6" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38627 } } peers { permanent_uuid: "f9478c21d299492fb7f6b216d2e42d97" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 43449 } } peers { permanent_uuid: "85e67a14487f4813aaff79b9be811a29" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } }
I20260619 08:23:27.944223 27464 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9478c21d299492fb7f6b216d2e42d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9478c21d299492fb7f6b216d2e42d97, State: Initialized, Role: FOLLOWER
I20260619 08:23:27.944298 27464 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9478c21d299492fb7f6b216d2e42d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c844c1883d416d80df3609e1dd7ea6" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38627 } } peers { permanent_uuid: "f9478c21d299492fb7f6b216d2e42d97" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 43449 } } peers { permanent_uuid: "85e67a14487f4813aaff79b9be811a29" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } }
I20260619 08:23:27.944620 27464 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9478c21d299492fb7f6b216d2e42d97 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:23:27.945577 27438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77c844c1883d416d80df3609e1dd7ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f9478c21d299492fb7f6b216d2e42d97"
I20260619 08:23:27.945689 27438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9478c21d299492fb7f6b216d2e42d97 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:23:27.946553 27438 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f9478c21d299492fb7f6b216d2e42d97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77c844c1883d416d80df3609e1dd7ea6 in term 1.
I20260619 08:23:27.947394 27437 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f9478c21d299492fb7f6b216d2e42d97 [term 1 FOLLOWER]: Refusing update from remote peer 77c844c1883d416d80df3609e1dd7ea6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:23:27.947649 27579 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9478c21d299492fb7f6b216d2e42d97" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:23:27.949280 27603 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9478c21d299492fb7f6b216d2e42d97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77c844c1883d416d80df3609e1dd7ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c844c1883d416d80df3609e1dd7ea6" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38627 } } peers { permanent_uuid: "f9478c21d299492fb7f6b216d2e42d97" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 43449 } } peers { permanent_uuid: "85e67a14487f4813aaff79b9be811a29" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } } }
I20260619 08:23:27.949474 27603 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9478c21d299492fb7f6b216d2e42d97 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:23:27.951918 27579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "77c844c1883d416d80df3609e1dd7ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c844c1883d416d80df3609e1dd7ea6" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38627 } } peers { permanent_uuid: "f9478c21d299492fb7f6b216d2e42d97" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 43449 } } peers { permanent_uuid: "85e67a14487f4813aaff79b9be811a29" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } } }
I20260619 08:23:27.952003 27579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6 [sys.catalog]: This master's current role is: LEADER
I20260619 08:23:27.951967 27599 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9478c21d299492fb7f6b216d2e42d97 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "77c844c1883d416d80df3609e1dd7ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c844c1883d416d80df3609e1dd7ea6" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38627 } } peers { permanent_uuid: "f9478c21d299492fb7f6b216d2e42d97" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 43449 } } peers { permanent_uuid: "85e67a14487f4813aaff79b9be811a29" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } } }
I20260619 08:23:27.952062 27599 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9478c21d299492fb7f6b216d2e42d97 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:23:27.952145 27581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77c844c1883d416d80df3609e1dd7ea6: Generated new TSK 0
I20260619 08:23:27.952307 27581 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:23:27.949605 27589 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9478c21d299492fb7f6b216d2e42d97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77c844c1883d416d80df3609e1dd7ea6. Latest consensus state: current_term: 1 leader_uuid: "77c844c1883d416d80df3609e1dd7ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c844c1883d416d80df3609e1dd7ea6" member_type: VOTER last_known_addr { host: "127.4.151.190" port: 38627 } } peers { permanent_uuid: "f9478c21d299492fb7f6b216d2e42d97" member_type: VOTER last_known_addr { host: "127.4.151.189" port: 43449 } } peers { permanent_uuid: "85e67a14487f4813aaff79b9be811a29" member_type: VOTER last_known_addr { host: "127.4.151.188" port: 36033 } } }
I20260619 08:23:27.953848 27589 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9478c21d299492fb7f6b216d2e42d97 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 08:23:27.953914 27606 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f9478c21d299492fb7f6b216d2e42d97: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:23:27.953963 27606 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 08:23:27.981292 27392 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:27.979690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60762 (local address 127.4.151.190:38627)
0619 08:23:27.979743 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:27.979746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:27.980583 (+ 837us) server_negotiation.cc:407] Connection header received
0619 08:23:27.980687 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:27.980689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:27.980701 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:27.980742 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:27.981220 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60762: BlockingRecv error: recv got EOF from 127.0.0.1:60762 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260619 08:23:27.982159 27469 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:27.980029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53410 (local address 127.4.151.189:43449)
0619 08:23:27.980067 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:27.980069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:27.981680 (+ 1611us) server_negotiation.cc:407] Connection header received
0619 08:23:27.981783 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:27.981785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:27.981795 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:27.981823 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:27.982103 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53410: BlockingRecv error: recv got EOF from 127.0.0.1:53410 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 08:23:27.982964 27544 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:27.980327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33866 (local address 127.4.151.188:36033)
0619 08:23:27.980384 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:27.980386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:27.982479 (+ 2093us) server_negotiation.cc:407] Connection header received
0619 08:23:27.982569 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:27.982570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:27.982580 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:27.982607 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:27.982906 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33866: BlockingRecv error: recv got EOF from 127.0.0.1:33866 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260619 08:23:27.985337 27469 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:27.984550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53412 (local address 127.4.151.189:43449)
0619 08:23:27.984583 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:27.984585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:27.984836 (+ 251us) server_negotiation.cc:407] Connection header received
0619 08:23:27.984933 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:27.984934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:27.984950 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:27.984977 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:27.985294 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53412: BlockingRecv error: recv got EOF from 127.0.0.1:53412 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 08:23:27.990537 27469 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:27.989739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53420 (local address 127.4.151.189:43449)
0619 08:23:27.989773 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:27.989775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:27.990019 (+ 244us) server_negotiation.cc:407] Connection header received
0619 08:23:27.990111 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:27.990112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:27.990122 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:27.990151 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:27.990496 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53420: BlockingRecv error: recv got EOF from 127.0.0.1:53420 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260619 08:23:27.994721 27469 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:27.993882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53434 (local address 127.4.151.189:43449)
0619 08:23:27.993915 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:27.993917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:27.994203 (+ 286us) server_negotiation.cc:407] Connection header received
0619 08:23:27.994303 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:27.994304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:27.994314 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:27.994345 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:27.994678 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53434: BlockingRecv error: recv got EOF from 127.0.0.1:53434 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 08:23:28.000720 27469 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:27.999902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53448 (local address 127.4.151.189:43449)
0619 08:23:27.999935 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:27.999937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:28.000171 (+ 234us) server_negotiation.cc:407] Connection header received
0619 08:23:28.000272 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:28.000273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:28.000283 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:28.000334 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:28.000673 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53448: BlockingRecv error: recv got EOF from 127.0.0.1:53448 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 08:23:28.011299 27469 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:28.010279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53464 (local address 127.4.151.189:43449)
0619 08:23:28.010323 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:28.010325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:28.010599 (+ 274us) server_negotiation.cc:407] Connection header received
0619 08:23:28.010729 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:28.010731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:28.010744 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:28.010778 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:28.011206 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53464: BlockingRecv error: recv got EOF from 127.0.0.1:53464 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 08:23:28.030066 27469 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:28.029008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53474 (local address 127.4.151.189:43449)
0619 08:23:28.029056 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:28.029058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:28.029352 (+ 294us) server_negotiation.cc:407] Connection header received
0619 08:23:28.029509 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:28.029511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:28.029525 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:28.029558 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:28.029990 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53474: BlockingRecv error: recv got EOF from 127.0.0.1:53474 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260619 08:23:28.064800 27469 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:28.063790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53478 (local address 127.4.151.189:43449)
0619 08:23:28.063836 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:28.063838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:28.064158 (+ 320us) server_negotiation.cc:407] Connection header received
0619 08:23:28.064268 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:28.064270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:28.064283 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:28.064315 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:28.064732 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53478: BlockingRecv error: recv got EOF from 127.0.0.1:53478 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260619 08:23:28.131093 27469 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:28.130483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53488 (local address 127.4.151.189:43449)
0619 08:23:28.130550 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:28.130552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:28.130586 (+ 34us) server_negotiation.cc:407] Connection header received
0619 08:23:28.130676 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:28.130677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:28.130687 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:28.130741 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:28.131043 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53488: BlockingRecv error: recv got EOF from 127.0.0.1:53488 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 08:23:28.261425 27469 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:28.260484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53502 (local address 127.4.151.189:43449)
0619 08:23:28.260579 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:28.260581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:28.260632 (+ 51us) server_negotiation.cc:407] Connection header received
0619 08:23:28.260758 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:28.260759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:28.260773 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:28.260852 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:28.261301 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53502: BlockingRecv error: recv got EOF from 127.0.0.1:53502 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 08:23:28.519340 27469 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:23:28.518642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53510 (local address 127.4.151.189:43449)
0619 08:23:28.518739 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:28.518741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:28.518759 (+ 18us) server_negotiation.cc:407] Connection header received
0619 08:23:28.518790 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:28.518790 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:28.518801 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:28.518867 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:23:28.519269 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53510: BlockingRecv error: recv got EOF from 127.0.0.1:53510 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 08:23:28.890008 27557 catalog_manager.cc:1259] Loaded cluster ID: 501bb741dc584839ad8e7648ba371b55
I20260619 08:23:28.890079 27557 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 85e67a14487f4813aaff79b9be811a29: loading cluster ID for follower catalog manager: success
I20260619 08:23:28.890951 27557 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 85e67a14487f4813aaff79b9be811a29: acquiring CA information for follower catalog manager: success
I20260619 08:23:28.891259 27557 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 85e67a14487f4813aaff79b9be811a29: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 08:23:28.954324 27606 catalog_manager.cc:1259] Loaded cluster ID: 501bb741dc584839ad8e7648ba371b55
I20260619 08:23:28.954396 27606 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f9478c21d299492fb7f6b216d2e42d97: loading cluster ID for follower catalog manager: success
I20260619 08:23:28.955189 27606 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f9478c21d299492fb7f6b216d2e42d97: acquiring CA information for follower catalog manager: success
I20260619 08:23:28.955448 27606 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f9478c21d299492fb7f6b216d2e42d97: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 08:23:29.036019 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:29.032844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53524 (local address 127.4.151.189:43449)
0619 08:23:29.032972 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:29.032975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:29.032989 (+ 14us) server_negotiation.cc:407] Connection header received
0619 08:23:29.033190 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:29.033192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:29.033287 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:29.033440 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:23:29.033610 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:29.034213 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:29.035086 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:29.035319 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:29.035403 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:29.035605 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:23:29.035735 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:23:29.035834 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:29.035879 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 08:23:29.035903 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260619 08:23:29.040956 27679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:29.038228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33876 (local address 127.4.151.188:36033)
0619 08:23:29.038348 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:29.038350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:29.038359 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:23:29.038398 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:29.038399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:29.038435 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:29.038506 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:23:29.038765 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:29.039311 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:29.040133 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:29.040354 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:29.040417 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:29.040584 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:23:29.040696 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:23:29.040755 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:29.040802 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 08:23:29.040820 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260619 08:23:29.045742 27679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:29.042924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33880 (local address 127.4.151.188:36033)
0619 08:23:29.042960 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:29.042961 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:29.043130 (+ 169us) server_negotiation.cc:407] Connection header received
0619 08:23:29.043190 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:29.043191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:29.043202 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:29.043261 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:23:29.043428 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:29.043916 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:29.045041 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:29.045238 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:29.045314 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:29.045477 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:23:29.045526 (+ 49us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:23:29.045573 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:29.045630 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 08:23:29.045651 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:23:29.046294 27686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:29.042829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60766 (local address 127.4.151.190:38627)
0619 08:23:29.042944 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:29.042947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:29.043083 (+ 136us) server_negotiation.cc:407] Connection header received
0619 08:23:29.043181 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:29.043182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:29.043239 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:29.043319 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:23:29.043500 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:29.044022 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:29.044996 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:29.045324 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:29.045401 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:29.045922 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:23:29.046036 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:23:29.046126 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:29.046182 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 08:23:29.046207 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":57,"threads_started":1}
I20260619 08:23:29.046945 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:23:29.043001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53540 (local address 127.4.151.189:43449)
0619 08:23:29.043042 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 08:23:29.043044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:23:29.043242 (+ 198us) server_negotiation.cc:407] Connection header received
0619 08:23:29.043339 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:23:29.043340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:23:29.043351 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:23:29.043430 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:23:29.044185 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:29.044911 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:29.046172 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:23:29.046431 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:23:29.046473 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:23:29.046696 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:23:29.046733 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:23:29.046784 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 08:23:29.046838 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 08:23:29.046862 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:23:29.053601 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 27321
I20260619 08:23:29.059037 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 27396
I20260619 08:23:29.063832 4702 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR4iS53/build/release/bin/kudu with pid 27471
2026-06-19T08:23:29Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2840 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5675 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (278470 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST